Electronic device for selecting image processing technique based on shape and operating method thereof

ABSTRACT

An electronic device and an operating method thereof are provided. The electronic device includes a camera module, a memory module, and a processor operatively coupled with the camera module and the memory module. The processor acquires an image through the camera module, extracts distance information based on the acquired image, determines an image processing technique for an object based on the extracted distance information, applies the determined image processing technique to the acquired image to generate a new image, and displays the new image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to a Korean Patent Application filed in the Korean Intellectual PropertyOffice on Feb. 19, 2016 and assigned Serial No. 10-2016-0020009, thedisclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

1. Field of the Disclosure

The present disclosure relates generally to an electronic device capableof processing an image, and an operating method thereof.

2. Description of Related Art

With the growth of digital technologies, electronic devices can be usedin various types such as mobile communication terminals, smart phones,tablet Personal Computers (PCs), Personal Digital Assistants (PDAs),electronic organizers, notebook computers, wearable devices, etc. Theelectronic devices are coming to a mobile convergence level encompassingeven functions of other devices. The electronic device can include acamera module. The electronic device can photograph an object imagethrough the camera module, and can store or transmit the photographedimage to an external another electronic device.

When processing a photographed image, the electronic device could set animage processing function that is in advance prepared through a menu,etc., and apply the set image processing function, thereby process theimage acquired through the camera module. For example, in case where theelectronic device uses an image filter, a user can previously set theimage filter, and the electronic device could apply a function of theset image filter to the image acquired through the camera module,thereby processing the image.

SUMMARY

An electronic device according to various example embodiments of thepresent disclosure can provide an apparatus and method capable ofextracting distance information from image data acquired, automaticallydetermining an image processing technique based on the extracteddistance information, and applying the determined image processingtechnique to the acquired image, thereby generating a new image.

An electronic device according to various example embodiments of thepresent disclosure can provide an apparatus and method capable ofextracting distance information from image data acquired through animage sensor, automatically determining an image filter based on theextracted distance information, and applying the determined image filterto the acquired image, thereby generating a new image.

An electronic device according to various example embodiments of thepresent disclosure can provide an apparatus and method capable ofextracting a contour line of image data generated through an imagesensor, to recognize configuration information of an object, setting animage filter based on the configuration information, and applying theimage filter to the image data, thereby generating a new image.

An electronic device according to various example embodiments of thepresent disclosure can include a camera module, a memory module, and aprocessor operatively coupled with the camera module and the memorymodule. The processor can acquire image data through the camera module,extract distance information based on the acquired image data, determinean image processing technique for an object based on the extracteddistance information, apply the determined image processing technique tothe acquired image data, and display the applied image data.

A method for operating in an electronic device according to variousexample embodiments of the present disclosure can include the operationsof acquiring image data through an image sensor, extracting distanceinformation from the acquired image data, determining an imageprocessing technique for an object based on the extracted distanceinformation, applying the determined image processing technique to theacquired image data, to generate a new image, and displaying the newimage.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other aspects, features and attendant advantages of thepresent disclosure will be more readily appreciated and understood fromthe following detailed description, taken in conjunction with theaccompanying drawings, in which like reference numerals refer to likeelements, and wherein:

FIG. 1 is a diagram illustrating an example network environment systemaccording to various example embodiments;

FIG. 2 is a block diagram illustrating an example electronic deviceaccording to various example embodiments;

FIG. 3 is a block diagram illustrating an example program moduleaccording to various example embodiments;

FIG. 4 is a block diagram illustrating an example electronic deviceaccording to various example embodiments of the present disclosure;

FIG. 5 is a block diagram illustrating an example apparatus forprocessing an acquired image and generating a new image in an electronicdevice according to various example embodiments of the presentdisclosure;

FIGS. 6A, 6B, 6C, 6D and 6E are diagrams illustrating example imagefilters according to various example embodiments of the presentdisclosure;

FIG. 7 is a diagram illustrating an example image sensor according tovarious example embodiments of the present disclosure.

FIG. 8 is a diagram illustrating an example pixel array of an imagesensor according to an example embodiment of the present disclosure;

FIGS. 9A, 9B and 9C are diagrams illustrating example structures of aunit pixel and sub pixels of an example image sensor according tovarious example embodiments of the present disclosure;

FIG. 10A and FIG. 10B are diagrams illustrating examples of a unit pixeland sub pixels of an image sensor according to various exampleembodiments of the present disclosure;

FIG. 11A and FIG. 11B are diagrams illustrating an example operation ofan image sensor including a top surface phase difference sensor;

FIG. 12A and FIG. 12B are diagrams extending and illustrating a sectionof one portion of an example image sensor according to an exampleembodiment of the present disclosure;

FIG. 13A and FIG. 13B are diagrams illustrating an example of displayingacquired image and distance information in an electronic deviceaccording to various example embodiments of the present disclosure;

FIGS. 14A, 14B, 14C and 14D are diagrams illustrating an example ofimage processing of an electronic device according to various exampleembodiments of the present disclosure;

FIG. 15 is a block diagram illustrating an example image processingdevice according to various example embodiments of the presentdisclosure; and

FIG. 16 is a flowchart illustrating an example method of processing animage in an electronic device according to various example embodimentsof the present disclosure.

DETAILED DESCRIPTION

Various example embodiments of the present disclosure are described ingreater detail with reference to the accompanying drawings. The same orsimilar components may be designated by the same or similar referencenumerals although they are illustrated in different drawings. Detaileddescriptions of constructions or processes known in the art may beomitted to avoid obscuring the subject matter of the present disclosure.The terms used herein are defined in consideration of functions of thepresent disclosure and may vary depending on a user's or an operator'sintension and usage. Therefore, the terms used herein should beunderstood based on the descriptions made herein. It is to be understoodthat the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise. In the presentdisclosure, an expression such as “A or B,” “at least one of A and B,”or “one or more of A and B” may include all possible combinations of thelisted items. Expressions such as “first,” “second,” “primarily,” or“secondary,” as used herein, may represent various elements regardlessof order and/or importance and do not limit corresponding elements. Theexpressions may be used for distinguishing one element from anotherelement. When it is described that an element (such as a first element)is “(operatively or communicatively) coupled” to or “connected” toanother element (such as a second element), the element can be directlyconnected to the other element or can be connected through anotherelement (such as a third element).

An expression “configured to (or set)” used in the present disclosuremay be used interchangeably with, for example, “suitable for,” “havingthe capacity to,” “designed to,” “adapted to,” “made to,” or “capableof” according to a situation. A term “configured to (or set)” does notonly refer to “specifically designed to” by hardware. In somesituations, the expression “apparatus configured to” may refer to asituation in which the apparatus “can” operate together with anotherapparatus or component. For example, a phrase “a processor configured(or set) to perform A, B, and C” may be a dedicated processor, ageneric-purpose processor (such as a Central Processing Unit (CPU) or anapplication processor) that can perform a corresponding operation byexecuting at least one software program stored at an exclusive processor(such as an embedded processor) for performing a corresponding operationor at a memory device.

An electronic device according to various example embodiments of thepresent disclosure, may be embodied as, for example, at least one of asmart phone, a tablet Personal Computer (PC), a mobile phone, a videophone, an e-book reader, a desktop PC, a laptop PC, a netbook computer,a workstation, a server, a Personal Digital Assistant (PDA), a PortableMultimedia Player (PMP), an MPEG 3 (MP3) player, a medical equipment, acamera, and a wearable device, or the like, but is not limited thereto.The wearable device can include at least one of an accessory type (e.g.,a watch, a ring, a bracelet, an ankle bracelet, a necklace, glasses, acontact lens, or a Head-Mounted-Device (HMD)), a fabric or clothingembedded type (e.g., electronic garments), a body attachable type (e.g.,a skin pad or a tattoo), and an implantable circuit, or the like, but isnot limited thereto. The electronic device may be embodied as at leastone of, for example, a television, a Digital Versatile Disc (DVD)player, an audio device, a refrigerator, an air-conditioner, a cleaner,an oven, a microwave oven, a washing machine, an air cleaner, a set-topbox, a home automation control panel, a security control panel, a mediabox (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), a game console(e.g., Xbox™, PlayStation™), an electronic dictionary, an electronickey, a camcorder, and an electronic frame, or the like, but is notlimited thereto.

In another example embodiment, the electronic device may be embodied asat least one of various medical devices (such as, various portablemedical measuring devices (a blood sugar measuring device, a heartbeatmeasuring device, a blood pressure measuring device, or a bodytemperature measuring device), a Magnetic Resonance Angiography (MRA)device, a Magnetic Resonance Imaging (MRI) device, a Computed Tomography(CT) device, a scanning machine, and an ultrasonic wave device), anavigation device, a Global Navigation Satellite System (GNSS), an EventData Recorder (EDR), a Flight Data Recorder (FDR), a vehicleinfotainment device, electronic equipment for ship (such as, anavigation device for ship and gyro compass), avionics, a securitydevice, a head unit for a vehicle, an industrial or home robot, a drone,an Automated Teller Machine (ATM) of a financial institution, a Point OfSales (POS) device of a store, and an Internet of Things (IoT) device(e.g., a light bulb, various sensors, a sprinkler device, a fire alarm,a thermostat, a street light, a toaster, sports equipment, a hot watertank, a heater, and a boiler), or the like, but is not limited thereto.

According to an example embodiment, the electronic device may beembodied as at least one of a portion of furniture,building/construction or vehicle, an electronic board, an electronicsignature receiving device, a projector, and various measuring devices(e.g., water supply, electricity, gas, or electric wave measuringdevice), or the like, but is not limited thereto. An electronic device,according to an example embodiment, can be a flexible electronic deviceor a combination of two or more of the foregoing various devices. Anelectronic device, according to an example embodiment of the presentdisclosure, is not limited to the foregoing devices may be embodied as anewly developed electronic device. The term “user”, as used herein, canrefer to a person using an electronic device or a device using anelectronic device (e.g., an artificial intelligence electronic device).

Referring initially to FIG. 1, an electronic device 101 resides in anetwork environment 100. The electronic device 101 can include a bus110, a processor (e.g., including processing circuitry) 120, a memory130, an input/output interface (e.g., including input/output circuitry)150, a display 160, and a communication interface (e.g., includinginterface circuitry) 170. The electronic device 101 may be providedwithout at least one of the components, or may include at least oneadditional component.

The bus 110 can include a circuit for connecting the components 120through 170 and delivering communication signals (e.g., control messagesor data) therebetween.

The processor 120 can include various processing circuitry, such as, forexample, and without limitation, one or more of a dedicated processor, aCPU, an application processor, and a Communication Processor (CP), orthe like. The processor 120, for example, can perform an operation ordata processing with respect to control and/or communication of at leastanother component of the electronic device 101.

The memory 130 can include a volatile and/or nonvolatile memory. Thememory 130, for example, can store commands or data relating to at leastanother component of the electronic device 101. According to anembodiment, the memory 130 can store software and/or a program 140. Theprogram 140 can include, for example, a kernel 141, middleware 143, anApplication Programming Interface (API) 145, and/or an applicationprogram (or “application”) 147. At least part of the kernel 141, themiddleware 143, or the API 145 can be referred to as an Operating System(OS). The kernel 141 can control or manage system resources (e.g., thebus 110, the processor 120, or the memory 130) used for performingoperations or functions implemented by the other programs (e.g., themiddleware 143, the API 145, or the application program 147).Additionally, the kernel 141 can provide an interface for controlling ormanaging system resources by accessing an individual component of theelectronic device 101 from the middleware 143, the API 145, or theapplication program 147.

The middleware 143, for example, can serve an intermediary role forexchanging data between the API 145 or the application program 147 andthe kernel 141 through communication. Additionally, the middleware 143can process one or more job requests received from the applicationprogram 147, based on their priority. For example, the middleware 143can assign a priority for using a system resource (e.g., the bus 110,the processor 120, or the memory 130) of the electronic device 101 to atleast one of the application programs 147, and process the one or morejob requests. The API 145, as an interface through which the application147 controls a function provided from the kernel 141 or the middleware143, can include, for example, at least one interface or function (e.g.,an instruction) for file control, window control, image processing, orcharacter control. The input/output interface 150, for example, candeliver commands or data inputted from a user or another external deviceto other component(s) of the electronic device 101, or output commandsor data inputted from the other component(s) of the electronic device101 to the user or another external device.

The display 160, for example, can include a Liquid Crystal Display(LCD), a Light Emitting Diode (LED) display, an Organic Light EmittingDiode (OLED) display, a MicroElectroMechanical Systems (MEMS) display,or an electronic paper display, or the like, but is not limited thereto.The display 160, for example, can display various contents (e.g., texts,images, videos, icons, and/or symbols) to the user. The display 160 caninclude a touch screen, for example, and receive touch, gesture,proximity, or hovering inputs by using an electronic pen or a user'sbody part.

The communication interface 170, for example, can set a communicationbetween the electronic device 101 and an external device (e.g., a firstexternal electronic device 102, a second external electronic device 104,or a server 106). For example, the communication interface 170 cancommunicate with the external device (e.g., the second externalelectronic device 104 or the server 106) over a network 162 throughwireless communication or wired communication.

The wireless communication, for example, can include cellularcommunication using at least one of Long Term Evolution (LTE),LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), WidebandCDMA (WCDMA), Universal Mobile Telecommunications System (UMTS),Wireless Broadband (WiBro), or Global System for Mobile Communications(GSM). The wireless communication 164 can include, for example, at leastone of Wireless Fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE),Zigbee, Near Field Communication (NFC), magnetic secure transmission,Radio Frequency (RF), and Body Area Network (BAN). The wirelesscommunication can include GNSS. The GNSS can include, for example,Global Positioning System (GPS), Global Navigation Satellite System(GLONASS), Beidou navigation satellite system (Beidou), or Galileo (theEuropean global satellite-based navigation system). Hereafter, the GPScan be interchangeably used with the GNSS. The wired communication, forexample, can include at least one of Universal Serial Bus (USB), HighDefinition Multimedia Interface (HDMI), Recommended Standard 232(RS-232), power line communications, and Plain Old Telephone Service(POTS). The network 162 can include a telecommunications network, forexample, at least one of computer network (e.g., LAN or WAN), Internet,and telephone network.

Each of the first and second external electronic devices 102 and 104 canbe of the same as or of a different type from that of the electronicdevice 101. According to embodiments of the present disclosure, all orpart of operations executed in the electronic device 101 can be executedby another electronic device or a plurality of electronic devices (e.g.,the electronic device 102 or 104, or the server 106). To perform afunction or service automatically or by request, instead of performingthe function or the service by the electronic device 101, the electronicdevice 101 can request at least part of a function relating thereto fromanother device (e.g., the electronic device 102 or 104, or the server106). The other electronic device (e.g., the electronic device 102 or104, or the server 106) can perform the requested function or anadditional function and send its result to the electronic device 101.The electronic device 101 can provide the requested function or serviceby processing the received result. In doing so, for example, cloudcomputing, distributed computing, or client-server computing techniquescan be used.

FIG. 2 is a block diagram illustrating an example electronic device 201according to an example embodiment of the present disclosure.

The electronic device 201, for example, can include all or part of theabove-described electronic device 101 of FIG. 1. The electronic device201 includes one or more processors (e.g., an AP) (e.g., includingprocessing circuitry) 210, a communication module (e.g., includingcommunication circuitry) 220, a Subscriber Identification Module (SIM)224, a memory 230, a sensor module 240, an input device (e.g., includinginput circuitry) 250, a display 260, an interface (e.g., includinginterface circuitry) 270, an audio module 280, a camera module 291, apower management module 295, a battery 296, an indicator 297, and amotor 298.

The processor 210, for example, may include various processing circuitryand can control a plurality of hardware or software components connectedto the processor 210, and also can perform various data processing andoperations by executing an OS or an application program. The processor210 can be implemented with a System on Chip (SoC), for example. Theprocessor 210 can further include a Graphic Processing Unit (GPU) and/oran image signal processor. The processor 210 may include at least part(e.g., a cellular module 221) of the components illustrated in FIG. 2.The processor 210 can load commands or data received from at least oneother component (e.g., a nonvolatile memory) into a volatile memory,process them, and store various data in the nonvolatile memory.

The communication module 220 can have the same or similar configurationto the communication interface 170 of FIG. 1. The communication module220 may include various communication circuitry, such as, for example,and without limitation, the cellular module 221, a WiFi module 223, aBluetooth (BT) module 225, a GNSS module 227, an NFC module 228, and anRF module 229. The cellular module 221, for example, can provide voicecall, video call, Short Message Service (SMS), or Internet servicethrough a communication network. The cellular module 221 can identifyand authenticate the electronic device 201 in a communication network byusing the SIM (e.g., a SIM card) 224. The cellular module 221 canperform at least part of a function that the processor 210 provides. Thecellular module 221 can further include a CP. At least some (e.g., twoor more) of the cellular module 221, the WiFi module 223, the BT module225, the GNSS module 227, and the NFC module 228 can be included in oneIntegrated Circuit (IC) or an IC package. The RF module 229, forexample, can transmit/receive a communication signal (e.g., an RFsignal). The RF module 229, for example, can include a transceiver, aPower Amp Module (PAM), a frequency filter, a Low Noise Amplifier (LNA),or an antenna. According to another embodiment, at least one of thecellular module 221, the WiFi module 223, the BT module 225, the GNSSmodule 227, and the NFC module 228 can transmit/receive an RF signalthrough an additional RF module. The SIM 224, for example, can include acard including a SIM or an embedded SIM, and also can contain uniqueidentification information (e.g., an Integrated Circuit Card Identifier(ICCID)) or subscriber information (e.g., an International MobileSubscriber Identity (IMSI)).

The memory 230 (e.g., the memory 130) can include at least one of aninternal memory 232 and/or an external memory 234. The internal memory232 can include at least one of, for example, a volatile memory (e.g.,Dynamic RAM (DRAM), Static RAM (SRAM), or Synchronous Dynamic RAM(SDRAM)), and a non-volatile memory (e.g., One Time Programmable ROM(OTPROM), Programmable ROM (PROM), Erasable and Programmable ROM(EPROM), Electrically Erasable and Programmable ROM (EEPROM), mask ROM,flash ROM, flash memory, hard drive, and solid state drive (SSD)). Theexternal memory 234 can include flash drive, for example, Compact Flash(CF), Secure Digital (SD), micro SD, mini SD, extreme digital (xD),Multi-Media Card (MMC), or memory stick. The external memory 234 can befunctionally or physically connected to the electronic device 201through various interfaces.

The sensor module 240 can, for example, measure physical quantities ordetect an operating state of the electronic device 201, and thus convertthe measured or detected information into electrical signals. The sensormodule 240 can include at least one of a gesture sensor 240A, a gyrosensor 240B, an atmospheric pressure sensor 240C, a magnetic sensor240D, an acceleration sensor 240E, a grip sensor 240F, a proximitysensor 240G a color sensor 240H (e.g., a Red, Green, Blue (RGB) sensor),a biometric sensor 240I, a temperature/humidity sensor 240J, anillumination (e.g., light) sensor 240K, and an Ultra Violet (UV) sensor240M. Additionally or alternately, the sensor module 240 can include anE-nose sensor, an Electromyography (EMG) sensor, an Electroencephalogram(EEG) sensor, an Electrocardiogram (ECG) sensor, an InfraRed (IR)sensor, an iris sensor, and/or a fingerprint sensor. The sensor module240 can further include a control circuit for controlling at least onesensor therein. The electronic device, as part of the processor 210 orindividually, can further include a processor configured to control thesensor module 240 and thus control the sensor module 240 while theprocessor 210 is sleeping.

The input device 250 can include various input circuitry, such as, forexample, and without limitation, at least one of a touch panel 252, a(digital) pen sensor 254, a key 256, and an ultrasonic input device 258.The touch panel 252 can use at least one of, for example, capacitive,resistive, infrared, and ultrasonic methods. Additionally, the touchpanel 252 can further include a control circuit. The touch panel 252 canfurther include a tactile layer to provide a tactile response to a user.The (digital) pen sensor 254 can include, for example, part of a touchpanel or a sheet for recognition. The key 256 can include, for example,a physical button, a touch key, an optical key, or a keypad. Theultrasonic input device 258 can detect ultrasonic waves from an inputmeans through a microphone 288 and check data corresponding to thedetected ultrasonic waves.

The display 260 (e.g., the display 160) can include at least one of apanel 262, a hologram device 264, a projector 266, and/or a controlcircuit for controlling them. The panel 262 can be implemented to beflexible, transparent, or wearable, for example. The panel 262 and thetouch panel 252 can be configured with one or more modules. The panel262 can include a pressure sensor (or a force sensor) for measuring apressure of the user touch. The pressure sensor can be integrated withthe touch panel 252, or include one or more sensors separately from thetouch panel 252. The hologram device 264 can show three-dimensionalimages in the air by using the interference of light. The projector 266can display an image by projecting light on a screen. The screen, forexample, can be placed inside or outside the electronic device 201.

The interface 270 can include various interface circuitry, such as, forexample, and without limitation, an HDMI 272, a USB 274, an opticalinterface 276, or a D-subminiature (D-sub) 278. The interface 270 can beincluded in, for example, the communication interface 170 of FIG. 1.Additionally or alternately, the interface 270 can include a MobileHigh-Definition Link (MHL) interface, a SD card/MMC interface, or anInfrared Data Association (IrDA) standard interface.

The audio module 280, for example, can convert sounds into electricalsignals and convert electrical signals into sounds. At least somecomponents of the audio module 280 can be included in, for example, theinput/output interface 150 of FIG. 1. The audio module 280 can processsound information input or output through a speaker 282, a receiver 284,an earphone 286, or the microphone 288. The camera module 291, as adevice for capturing still images and videos, can include one or moreimage sensors (e.g., a front sensor or a rear sensor), a lens, an ImageSignal Processor (ISP), or a flash (e.g., an LED or a xenon lamp). Thepower management module 295, for example, can manage the power of theelectronic device 201. According to an embodiment of the presentdisclosure, the power management module 295 can include a PowerManagement IC (PMIC), a charger IC, or a battery or fuel gauge, forexample. The PMIC can have a wired and/or wireless charging method. Thewireless charging method can include, for example, a magnetic resonancemethod, a magnetic induction method, or an electromagnetic method, andcan further include an additional circuit for wireless charging, forexample, a coil loop, a resonant circuit, or a rectifier circuit. Thebattery gauge can measure the remaining capacity of the battery 296, ora voltage, current, or temperature of the battery 296 during charging.The battery 296 can include, for example, a rechargeable battery and/ora solar battery.

The indicator 297 can display a specific state of the electronic device201 or part thereof (e.g., the processor 210), for example, a bootingstate, a message state, or a charging state. The motor 298 can convertelectrical signals into mechanical vibration and generate a vibration orhaptic effect. The electronic device 201 can include a mobile TVsupporting device (e.g., a GPU) for processing media data according tostandards such as Digital Multimedia Broadcasting (DMB), Digital VideoBroadcasting (DVB), or MediaFLOW™. Each of the above-describedcomponents of the electronic device can be configured with at least onecomponent and the name of a corresponding component can vary accordingto the kind of an electronic device. According to an embodiment of thepresent disclosure, an electronic device (e.g., the electronic device201) can be configured to include at least one of the above-describedcomponents or an additional component, or to not include some of theabove-described components. Additionally, some of components in anelectronic device are configured as one entity, so that functions ofprevious corresponding components are performed identically.

FIG. 3 is a block diagram illustrating an example program moduleaccording to an embodiment of the present disclosure. A program module310 (e.g., the program 140) can include an OS for controlling a resourcerelating to an electronic device (e.g., the electronic device 101)and/or various applications (e.g., the application program 147) runningon the OS. The OS can include, for example, Android™, iOS™, Window™,Symbian™, Tizen™, or Bala™. Referring to FIG. 3, the program module 310can include a kernel 320 (e.g., the kernel 141), a middleware 330 (e.g.,the middleware 143), an API 360 (e.g., the API 145), and/or anapplication 370 (e.g., the application program 147). At least part ofthe program module 310 can be preloaded on an electronic device or canbe downloaded from an external electronic device (e.g., the electronicdevice 102, 104, or the server 106).

The kernel 320 includes, for example, at least one of a system resourcemanager 321 and/or a device driver 323. The system resource manager 321can control, allocate, or retrieve a system resource. According to anembodiment, the system resource manager 321 can include a processmanagement unit, a memory management unit, or a file system managementunit. The device driver 323 can include, for example, a display driver,a camera driver, a Bluetooth driver, a sharing memory driver, a USBdriver, a keypad driver, a WiFi driver, an audio driver, or anInter-Process Communication (IPC) driver.

The middleware 330, for example, can provide a function commonlyrequired by the application 370, or can provide various functions to theapplication 370 through the API 360 in order to allow the application370 to efficiently use a limited system resource inside the electronicdevice. The middleware 330 includes at least one of a runtime library335, an application manager 341, a window manager 342, a multimediamanager 343, a resource manager 344, a power manager 345, a databasemanager 346, a package manager 347, a connectivity manager 348, anotification manager 349, a location manager 350, a graphic manager 351,and a security manager 352.

The runtime library 335 can include, for example, a library module usedby a complier to add a new function through a programming language whilethe application 370 is running. The runtime library 335 can manageinput/output, manage memory, or arithmetic function processing. Theapplication manager 341, for example, can manage the life cycle of theapplications 370. The window manager 342 can manage a GUI resource usedin a screen. The multimedia manager 343 can recognize a format forplaying various media files and encode or decode a media file by usingthe codec in a corresponding format. The resource manager 344 can managea source code of the application 3740 or a memory space. The powermanager 345 can manage the capacity or power of the battery and providepower information for an operation of the electronic device. The powermanager 345 can operate together with a Basic Input/Output System(BIOS). The database manager 346 can create, search, or modify adatabase used in the application 370. The package manager 347 can manageinstallation or updating of an application distributed in a package fileformat.

The connectivity manger 348 can manage, for example, a wirelessconnection. The notification manager 349 can provide an event, such asincoming messages, appointments, and proximity alerts, to the user. Thelocation manager 350 can manage location information of an electronicdevice. The graphic manager 351 can manage a graphic effect to beprovided to the user or a user interface relating thereto. The securitymanager 352 can provide, for example, system security or userauthentication. The middleware 330 can include a telephony manager formanaging a voice or video call function of the electronic device, or amiddleware module for combining various functions of the above-describedcomponents. The middleware 330 can provide a module specialized for eachtype of OS. The middleware 330 can dynamically delete part of theexisting components or add new components. The API 360, as a set of APIprogramming functions, can be provided as another configurationaccording to the OS. For example, Android or iSO can provide one API setfor each platform, and Tizen can provide two or more API sets for eachplatform.

The application 370 can include at least one of a home 371, a dialer372, an SMS/Multimedia Messaging System (MIMS) 373, an Instant Message(IM) 374, a browser 375, a camera 376, an alarm 377, a contact 378, avoice dial 379, an e-mail 380, a calendar 381, a media player 382, analbum 383, a clock 384, health care (e.g., measure an exercise amount orblood sugar level), or environmental information (e.g., air pressure,humidity, or temperature information) provision application. Theapplication 370 can include an information exchange application forsupporting information exchange between the electronic device and anexternal electronic device. The information exchange application caninclude, for example, a notification relay application for relayingspecific information to the external device or a device managementapplication for managing the external electronic device. For example,the notification relay application can relay notification informationfrom another application of the electronic device to an externalelectronic device, or receive and forward notification information froman external electronic device to the user. The device managementapplication, for example, can install, delete, or update a function(e.g., turn-on/turn off of the external electronic device itself (orsome components) or display brightness (or resolution) adjustment) of anexternal electronic device communicating with the electronic device, oran application operating in the external electronic device. Theapplication 370 can include a specified application (e.g., a health careapplication of a mobile medical device) according to a property of theexternal electronic device. The application 370 can include anapplication received from an external electronic device. At least partof the program module 310 can be implemented (e.g., executed) withsoftware, firmware, hardware (e.g., the processor 210), or a combinationof at least two of them, and include a module, a program, a routine, aset of instructions, or a process for executing one or more functions.

A term “module” used in the present disclosure includes a unit includinghardware, software, or firmware, or any combination thereof, and may beinterchangeably used with a term such as a unit, a logic, a logicalblock, a component, a circuit, and the like. The “module” may be anintegrally constructed component or a minimum unit or one part thereoffor performing one or more functions. The “module” may be mechanicallyor electrically implemented, and may include, for example, and withoutlimitation, a dedicated process, a CPU, an Application-SpecificIntegrated Circuit (ASIC) chip, a Field-Programmable Gate Arrays(FPGAs), or a programmable-logic device, which is known or to bedeveloped to perform certain operations.

At least one part of an apparatus (e.g., modules or functions thereof)or method (e.g., operations) according to various example embodimentsmay be implemented with an instruction stored in a computer-readablestorage media (e.g., the memory 130). If the instruction is executed byone or more processors (e.g., the processor 120), the one or moreprocessors may perform a function corresponding to the instruction. Thecomputer-readable storage media may include a hard disk, a floppy disk,magnetic media (e.g., a magnetic tape), optical media (e.g., a CompactDisc-ROM (CD-ROM), a Digital Versatile Disc (DVD), magnetic-optic media(e.g., a floptical disk)), an internal memory, or the like. Theinstruction may include a code created by a compiler or a codeexecutable by an interpreter.

The module or programming module according to various exampleembodiments may further include at least one or more elements among theaforementioned elements, or may omit some of them, or may furtherinclude additional other elements. Operations performed by a module,programming module, or other elements may be executed in a sequential,parallel, repetitive, or heuristic manner. In addition, some of theoperations may be executed in a different order or may be omitted, orother operations may be added.

In various example embodiments of the present disclosure, the term“image information” may, for example, be used as the term includingimage data output from an image sensor and distance information betweenan electronic device and an object. The “image data” may include, forexample, a color image that is generated in the image sensor. The“distance information” may include, for example, distance informationbetween the object and the electronic device, which is generated in theimage sensor. The distance information can be generated, for example,through phase difference information between pixels of the image sensor.For example, a 2 Photo Detector (2-PD) image sensor or 4-PD image sensorcan determine the distance information through a phase differencebetween sub pixels. The image sensor including a top surface phasedifference sensor can determine the distance information based on anoutput of the top surface phase difference sensor. The “imageinformation” may include, for example, “three-Dimensional (3D)information”. “Configuration information” may include, for example,information that can be estimated based on a depth map of an objectincluded in an image. The configuration information can include a rangeof the object and/or object position information within the image. Theconfiguration information can further include photographing postureinformation of the electronic device. A “new image” may include, forexample, an image that is generated by applying an image processingtechnique, which may be automatically set in the electronic device, toan acquired image. The image processing technique may include an imagefiltering method, for example.

FIG. 4 is a block diagram illustrating an example electronic deviceaccording to various example embodiments of the present disclosure.

Referring to FIG. 4, the electronic device can include a processor(e.g., including processing circuitry) 400, a memory module 410, acamera module (e.g., including a camera and associated image obtainingand processing circuitry) 420, a sensor module 430, an input module(e.g., including input circuitry) 440, and/or a display module 450. Insome example embodiment, the electronic device can omit at least one ofthe elements or additionally include another element.

The processor 400 may include various processing circuitry and canexecute operation or data processing according to control and/orapplication execution of at least one another element of the electronicdevice. The processor 400 can analyze image data acquired in aphotographing mode, to extract distance information, and process animage based on the extracted distance information. The processor 400 canset a photographing condition. The processor 400 can recognizeconfiguration information of objects within image data based on theimage data that is acquired based on the photographing condition, andset an image processing technique (for example, image filter) based onthe configuration information of the object, and apply the set imageprocessing technique to the acquired image data, thereby generating anew image.

The memory module 410 can include a volatile and/or non-volatile memory.The memory module 410 can store a command or data related to at leastanother element of the electronic device. The memory module 410 canstore software and/or program. The program can, for example, include akernel, a middleware, an Application Programming Interface (API), anapplication program (or “application”), etc. At least some of thekernel, the middleware, or the API can be called an Operating System(OS). The memory module 410 according to various example embodiments ofthe present disclosure can store image filters.

The camera module 420 can include various image receiving and processingcircuitry and element, such as, for example, and without limitation, alens and an image sensor, and can acquire an image including an object.The camera module 420 can be the camera module 291 of FIG. 2. The cameramodule 420 can be called an imaging device as well. The image sensor cangenerate image data. The image data can include a signal for extractingdistance information (e.g., depth information). For example, the imagedata can include pixel image data and a signal for extracting(determining) distance information. The image sensor can have an imagesensor construction (for example, 2-PD image sensor, 4-PD image sensor,etc.) in which one unit pixel includes a plurality of sub pixels. Theimage sensor can have a construction that includes a pixel sensor andtop surface phase difference sensors. The image sensor can be a Time OfFlight (TOF) image sensor or structured-light image sensor that canextract distance information between the electronic device and theobject. The image sensor can include an IR or ultrasonic measurementdevice, etc., and output a signal for extracting distance information ofthe object.

The sensor module 430 can include various sensors capable of sensinginformation of a motion of the electronic device, a posture thereof,etc. The sensor module 430 can be the sensor module 240 of FIG. 2. Thesensor module 430 can include a tilt sensor, an acceleration sensor, agyro sensor, etc. The sensor module 430 can further include some or allof a pressure sensor, a proximity sensor, a barometer, a terrestrialmagnetism sensor (or compass sensor), an ultrasonic sensor, an opticalflow sensing movement by using an image, a temperature-humidity sensor,an illuminance sensor, a UV sensor, and/or a gesture sensor.

The sensor module 430 according to various example embodiments of thepresent disclosure can recognize at least one of a tilting of theelectronic device, a movement thereof, and/or a grasping thereof in aphotographing mode. The tilt sensor can sense the tilting of theelectronic device. The tilt sensor can be replaced with the accelerationsensor and/or the gyro sensor as well.

The input module 440 may include various input circuitry, and mayinclude, for example, the entire or partial construction of the inputoutput interface 150 of FIG. 1 and the input device 250 of FIG. 2. Theinput module 440 can input an input and data for controlling anoperation of the electronic device. The input module 440 can be a touchpanel. The input module 440 can include a (digital) pen sensor. Theinput module 440 can include key buttons.

The display module 450 can be the display 160 of FIG. 1 or the display260 of FIG. 2. The display module 450 can be a Liquid Crystal Display(LCD), or a Light Emitting Diode (LED) display. The LED display caninclude an Organic LED (OLED) and an Active Matrix OLED (AMOLED), or thelike.

The input module 440 and the display module 450 can include a touchscreen. The touch screen can display a screen under the control of theprocessor 400, and can detect a touch, gesture, proximity, or hoveringinput using an electronic pen or a part of the user's body.

The electronic device according to various example embodiments of thepresent disclosure can further include a communication module (not shownin FIG. 4). The communication module can include various communicationcircuitry, including, for example, and without limitation, at least oneof a wireless communication module and a wired communication module. Thewireless communication module can include some or all of an RF module, acellular module, a WiFi module, a Bluetooth module, and/or a GPS module.

The cellular module can use at least one of Long Term Evolution (LTE),LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), WidebandCDMA (WCDMA), Universal Mobile Telecommunications System (UNITS),Wireless Broadband (WiBro), Global System for Mobile Communications(GSM), etc.

Also, the communication module can include at least one of a WiFimodule, a Bluetooth (BT) module, Near Field Communication (NFC), aGlobal Navigation Satellite System (GNSS), a GPS, etc. In accordancewith a use area, a bandwidth, etc., the GNSS can, for example, includeat least one of a GPS, a Global navigation satellite system (Glonass), aBeidou navigation satellite system (hereinafter, “Beidou”), Galileo, orthe European global satellite-based navigation system. The “GNSS” of thecommunication module can be used interchangeably with the “GPS”.

FIG. 5 is a diagram illustrating an example apparatus for processing anacquired image and generating a new image in an electronic deviceaccording to various example embodiments of the present disclosure.

Referring to FIG. 5, the electronic device can include a camera module510, a photographing setting module 520, an information acquisitionmodule 530, a filter determination module 540, a filter applying module550, and/or a display module 560.

The camera module 510 can be the camera module 420 of FIG. 4. The cameramodule 510 may include various circuitry and elements that can convertan optical signal into an electrical signal, and output the electricalsignal as image data. The camera module 510 can include an image sensor.The image sensor can acquire image data from an optical signal that isincident through a lens.

The image sensor can include a one-piece type image sensor capable ofgenerating an image and a signal for extracting distance information,and a stereo type image sensor generating each of an image and a signalfor extracting distance information.

According to an example embodiment, the one-piece type image sensor canbe an image sensor (for example, 2-PD image sensor, 4-PD image sensor,etc.) in which one unit pixel has a plurality of sub pixels, an imagesensor comprising a top surface phase difference sensor and pixelsensors, etc. The one-piece type image sensor can output the signal forextracting the distance information and the image data, at the sametime. The image data and the signal for extracting the distanceinformation that are output from the image sensor can be mapped to eachother.

According to an example embodiment, a stereo type image sensor canindependently construct each of a color sensor array generating colordata and a depth sensor array outputting a signal for calculating(determining) distance information. For example, the depth sensor arraycan be an IR sensor array. The stereo type image sensor canindependently generate each of a color image and a signal for extractingdistance information. For example, the stereo type image sensor cangenerate the color image and the signal for extracting the distanceinformation at mutually different time.

The image sensor of the camera module 420 according to various exampleembodiments of the present disclosure can be the one-piece type imagesensor including all of sensors generating an image and a signal fordetermining distance information. The image sensor of the camera module420 according to various example embodiments of the present disclosurecan be the stereo type image sensor individually generating an image anda signal for determining distance information as well.

The image sensor according to various example embodiments of the presentdisclosure can generate image data that includes an image and a signalfor determining distance information. The image and the signal fordetermining the distance information can be generated at the same time,or the image and the signal for determining the distance information canbe generated individually. Also, the image and the signal fordetermining the distance information can be mapped by pixel or by setregion, or may not be mapped.

The photographing setting module 520 can include various circuitryand/or program elements configured to set a general photographingcondition such as a focus of the camera module 510, light exposure,white balance, etc. The photographing setting module 520 can set theapplying or non-applying of an image processing technique to imagesacquired from the camera module 510. For example, the image processingtechnique can be image filtering. If a photographing operation applyingan image filter is set, the photographing setting module 520 can set aphotographing condition that is set in the camera module 510, and canset to determine the image filter in the filter determination module540.

The information acquisition module 530 can include various circuitryand/or program elements configured to get information such as a positionrelationship between objects that construct the field, a positionrelationship between the object and the electronic device, aphotographing posture of the electronic device, etc. For example, theinformation acquisition module 530 can sense a signal capable ofextracting distance information from image data that is acquired in thecamera module 510, and acquire (e.g., determine) distance information ofobjects.

The information acquisition module 530 can include various circuitryand/or program elements configured to be aware of a positionrelationship between the electronic device and the objects, based on theacquired distance information (e.g., depth map). The distanceinformation can be acquired using an image sensor (for example, 2-PDimage sensor or 4-PD image sensor) having a plurality of sub pixeldetection elements (e.g., photo diodes or photo detectors (PD)) as theunit extracting one pixel image, an image sensor including a top surfacephase difference sensor, a TOF image sensor, a structured-light imagesensor, a focus-adjustment image sensor, an IR or ultrasonic imagesensor, etc. According to an example embodiment, in case where it is the2-PD image sensor, the information acquisition module 530 can determinea phase difference between signals of two sub pixels within a unitpixel, to acquire a signal for extracting distance information.According to an example embodiment, in case where it is the image sensorincluding the top surface phase difference sensors, the informationacquisition module 530 can acquire distance information based on signalsof the top surface phase difference sensors within the image sensor. Thedistance information acquired in the information acquisition module 530can be a depth map.

According to an example embodiment, the image sensor of the cameramodule 510 can be a two-Dimensional (2D) image sensor generating imagedata. For example, the image sensor has a color pixel sensor array, andmay not generate a signal for extracting distance information. In casewhere the image sensor is the two-dimensional image sensor, theinformation acquisition module 530 can extract (e.g., image matching) anedge or boundary of image data generated in the image sensor. The edgeor boundary of the image data can be extracted based on a contour lineof an image, a change of color data, etc.

The information acquisition module 530 can acquire photographing postureinformation of the electronic device. For example, when a user performsa photographing operation through the camera module 510, the informationacquisition module 530 can sense a posture of the electronic device anda change of the posture from an output of the acceleration sensor and/orgyro sensor of the sensor module 430.

Based on an image output in the camera module 510 and/or distanceinformation (e.g., depth map) output in the information acquisitionmodule 530, the filter determination module 540 can determineconfiguration information of an object constructing the image. Theconfiguration information of the object can include a position of theobject in an image range (e.g., photo range), a size of the object,depth information between the object and the electronic device, etc.Based on the configuration information of the object, the filterdetermination module 540 can set an image processing technique suitableto the object within the image. For example, the image processingtechnique can be an image filter. The set image filter can be one imagefilter or a plurality of image filters.

The filter determination module 540 can include various circuitry and/orprogram elements, such as, for example, and without limitation, arecognition module and a filter determination module. The recognitionmodule recognizes configuration information of a main object based onimage data output in the camera module 510 and/or distance information(e.g., depth map) output in the information acquisition module 530. Thefilter determination module automatically may automatically set an imagefilter based on the recognized configuration information.

According to an example embodiment, the recognition module of the filterdetermination module 540 can recognize configuration information of anobject, based on distance information (e.g., depth map) output in theinformation acquisition module 530. For example, the recognition moduleof the filter determination module 540 can recognize the configurationinformation of the object, based on image data output in the cameramodule 510 and distance information output in the informationacquisition module 530.

According to an example embodiment, if the image sensor of the cameramodule 510 is an image sensor (for example, stereo type image sensor)generating a signal for extracting distance information and image dataat mutually different time, the recognition module of the filterdetermination module 540 can synchronize frames of the image data andthe signal extracting the distance information and perform a recognitionoperation.

According to an example embodiment, when the signal for extracting thedistance information and the image data are not mapped by pixel in theimage sensor of the camera module 510, the recognition module of thefilter determination module 540 can map the image data and the signalfor extracting the distance information in a set form (for example, mapa region of the image data and a region generating the signal forextracting the distance information) and perform a recognitionoperation.

The filter applying module 550 can process image data acquired in thecamera module 510 based on the image processing technique determined inthe filter determination module 540. If the image processing techniqueis an image filtering method, the filter applying module 550 can accessdata corresponding to an image filter determined in the filterdetermination module 540, through the memory module 410. And, the filterapplying module 550 can apply the set image filter to the image dataacquired in the camera module 510, to generate a new image.

The display module 560 can display an image (for example, a new imageapplying an image filter to an image) applied by the filter applyingmodule 550.

An image displayed in the display module 560 can be a preview image (orlive image). The electronic device can display an image acquired in thecamera module 510, as a preview image in the display module 560. If acapture request is generated, the electronic device can store in thememory module 410 an image acquired in the camera module 510. Forexample, if a photographing operation applying an image filter isrequested, the photographing setting module 520 can set a photographingmode, and the filter determination module 540 can set an image filter.The camera module 510 can generate image data including a main object.The image data can include a signal capable of extracting distanceinformation between the electronic device and objects, and image data ofthe object. The information acquisition module 530 can extract distanceinformation of an object from image data output in the camera module510, to generate the distance information (e.g., depth map) between theelectronic device and the objects. The information acquisition module530 can extract photographing posture information of the electronicdevice, based on an output of the sensor module 430 of FIG. 4.

The filter determination module 540 can recognize the setting of animage processing technique (for example, image filtering method) by anoutput of the photographing setting module 520. The filter determinationmodule 540 can recognize a feature of an object (e.g., a constituentelement of the object included in an image, a position relationship,etc.) by an output of the information acquisition module 530. Based onthe recognized object feature, the information acquisition module 530can extract range information (e.g., depth map) of the object. Based onthis information, the information acquisition module 530 can determine afilter that will be applied to an image. The filter applying module 550can apply the set image filter determined in the filter determinationmodule 540 to the image acquired in the camera module 510, to generate anew image. The new image can be an image applying the set image filterto the acquired image.

The display module 560 can display a new image that is generated in thefilter applying module 550. The new image displayed on the displaymodule 560 may be an image applying an image filter that is determinedbased on a distance between the object and the electronic device. And,the new image can be displayed as a preview image. The display module560 can display posture information of the electronic device that isacquired in the information acquisition module 560.

A user can check a new image displayed in the display module 560 andcorrect a posture of the electronic device. Also, based on the posturecorrection of the user, the information acquisition module 530 canacquire distance information of the object included in the image andposture information of the electronic device. The filter determinationmodule 540 can determine range information (e.g., depth map) of a newimage in accordance with the distance information and postureinformation that are corrected by an output of the informationacquisition module 530.

If a desired image is displayed in the display module 560, the user cangenerate a capture command. If the capture command is generated, theprocessor 400 can store in the memory module 410 a new image displayedin the display module 510. The image stored in the memory module 410 canbe an image to which a filter is applied. The image stored in the memorymodule 410 can include all of an image acquired in the camera module 510and the image to which the filter is applied.

FIGS. 6A, 6B, 6C, 6D and 6E are diagrams illustrating image filtersaccording to various example embodiments of the present disclosure.

FIG. 6A is a diagram illustrating an example characteristic of anexample linear filter. The linear filter can be a filter in which afocused point gets clear and, as it is farther away from the focusedpoint, a blur gets heavier. The blur can have the same-level blur in thedirection of one axis. A blur level can change in a direction verticalto the axis. The change of the blur level can be different according toa posture (e.g., tilting) of an electronic device (for example, camera).The linear filter can process to make clear the focused point in animage, and increase a blur intensity as it is farther away from a focus.For example, the size of a filter mask increases or the form of a maskcoefficient becomes different.

The linear filter can include a linear filter for emphasizing an object,a linear filter capable of blurring the background, and/or a linearfilter capable of emphasizing the object and blurring the background.

FIG. 6B is a diagram illustrating an example characteristic of anexample circular filter (or radial filter or polygonal filter). To catcheyes of a viewer who views an image, the circular filter can emphasize aspecific object. For example, the circular filter can increase anexposure and definition for an internal region of a corresponding figuresuch that attention can be paid to an object based on a shape of acircular form (for example, circle, oval or polygon) around the object.The circular filter can include a circular filter capable of blurring afocus of the background of an image, a circular filter capable ofemphasizing an object, or a circular filter capable of blurring thebackground and emphasizing the object.

FIG. 6C is a diagram illustrating an example characteristic of anexample lighting filter (or lighting effect). The lighting filter canapply several lighting effects to a Red, Green, and Blue (RGB) image.The lighting filter can produce and store a three-Dimensional (3D)effect through a texture of a grey shadow filter that is called a bumpmap, and use the stored 3D effect for another image as well. Thelighting filter can use a light source. The lighting filter can adjust alighting range, and can change a direction as well. The lighting filtercan adjust a lighting intensity, the lighting range, a lightingexposure, etc.

FIG. 6D is a diagram illustrating an example characteristic of anexample macro filter. Macro photographing can be photographing thatfills the entire image frame with an object by close-up photographingthe object. For example, the macro photographing increases an amount ofout-focus as it gets closer to the object, and can acquire an image thatout-focus processes all of the foreground and the background. The macrophotographing can be free to change a photographing angle andcomposition and thus, can be suitable to photographing a small thing.For example, the macro photographing can be a method of photographing ata distance very close to an object. The attraction of the macrophotographing is to be aware of the micro world that cannot be seen withnaked eyes, by close-up photographing the object.

In various example embodiments of the present disclosure, if a distancewith an object is sensed to be large, the electronic device can performa zooming operation to macro photograph the object. The electronicdevice can apply the macro filter to a macro photographed image, toemphasize the object. For example, if the circular filter is determinedas a result of analyzing distance information between the electronicdevice and the object and it is recognized that a distance with theobject is distant, the electronic device can perform the zoomingoperation to acquire an object image, and set the circular filter andthe macro filter to process the image. The macro filter can include amacro filter for emphasizing an object, a macro filter capable ofblurring the background, or a macro filter capable of emphasizing theobject and blurring the background.

FIG. 6E is a diagram illustrating an example characteristic of anexample selective focus (or narrow Depth Of Field (DOF)) filter. Whenprocessing an image, if the selective focus filter produces to blur theforeground and/or background of an object, the selective focus filtercan have an effect in which the object is more emphasized.

The electronic device according to various example embodiments of thepresent disclosure can acquire distance information (e.g., depth map)based on an acquired image, and recognize a range of an object based onthe distance information, and automatically determine an imageprocessing technique (for example, image filtering method) based on therecognized range of the object. The image processing technique may, forexample, determine at least one image filter among image filters of FIG.6A to FIG. 6E and apply the determined image filter to the acquiredimage, thereby generating a new image. The camera module can generatedistance information of objects that construct an image. An image sensorof the camera module can be a multi PD image sensor in which a unitpixel has a multi sub pixel construction, an image sensor including atop surface phase difference sensor, a TOF image sensor determining adistance (e.g., depth) by measuring the time taken for an output opticalsignal to reflect and return, a structured-light image sensor, afocus-adjustment image sensor, an IR image sensor, and/or an ultrasonicimage sensor.

FIG. 7 is a diagram illustrating an example image sensor according tovarious example embodiments of the present disclosure.

Referring to FIG. 7, the electronic device can include the processor400, the image sensor 700, and the display module 450. The image sensor700 can include a pixel array 730, a row driver 733, a readout driver735, and/or a timing signal generator 740.

According to various example embodiments, the image sensor 700 cangenerate an incident optical signal as image data. The processor 400 canprocess the generated image data and display the processed image data onthe display 160 that is operatively coupled with the electronic device.The processor 400 can check a mode for processing the image dataacquired in the image sensor 700, and process the image data based onthe checked mode.

The pixel array 730 can sense an object 710 that is captured through alens 720. The pixel array 730 can convert an optical signal of the lens720 into an electrical signal and generate image data. The timing signalgenerator 740 can generate a timing signal for activating row linepixels of the pixel array 730 through the row driver 733. And, thetiming signal generator 740 can generate a timing signal for reading-outcolumn line pixels of the pixel array 730. The pixel array 730 canactivate the row line pixels by the row driver 733, and can read-out thecolumn line pixel signals by the readout driver 735. In FIG. 7, it isillustrated that the timing signal generator 740 is included in theimage sensor 700, but can be included in the processor 400.

The pixel array 730 can be an array of pixels that include a micro lens,a color filter, and photo diodes. The pixel array 730 can include aplurality of unit pixels, and each unit pixel can include a plurality ofsub pixels. The sub pixel can include the photo diode. The image sensor700 can, for example, include a structure in which one unit pixel has atleast two or more sub pixels (e.g., photo diodes (PD)). The image sensor700 can output color information including at least one piece of colorinformation among Red (R), Green (G), and Blue (B).

FIG. 8 is a diagram illustrating an example pixel array of an imagesensor according to an example embodiment of the present disclosure.

Referring to FIG. 8, the pixel array 730 can include a plurality of unitpixels 810 to 840 that are arrayed in a matrix form. Each of the unitpixels 810 to 840 can include a plurality of sub pixels. The pixel array730 can at one time output signals of as many sub pixel levels as amultiplication of the number of the entire unit pixels and the number ofsub pixels of each unit pixel. The pixel array 730 can also sum upsignals of sub pixel levels of one unit pixel and output as many signalsas the number of the entire unit pixels The unit pixel can include atleast two sub pixels. FIG. 8 illustrates an example in which one unitpixel includes four sub pixels. In FIG. 8, a 1st unit pixel 810 caninclude four sub pixels 811 to 814, and a 2nd unit pixel 820 can includefour sub pixels 821 to 824, and a 3rd unit pixel 830 can include foursub pixels 831 to 834, and a 4th unit pixel 840 can include four subpixels 841 to 844 as well.

A sub-pixel pitch of the pixel array 730 according to various exampleembodiments of the present disclosure can be smaller than a pixel pitchof a common pixel array. FIG. 8 illustrates that the unit pixels 810 to840 include four sub pixels, respectively, but is not limited to this.The unit pixels can include two sub pixels, respectively. For example,the 1st unit pixel 810 can include two sub pixels 811 and 812, and the2nd unit pixel 820 can include two sub pixels 821 and 822, and the 3rdunit pixel 831 can include two sub pixels 831 and 832, and the 4th unitpixel 840 can include two sub pixels 841 to 842.

According to an example embodiment, the timing signal generator 740 cangenerate a timing signal for driving a row line and a column line of thepixel array 730 by the row driver 733 and the readout driver 735. Thepixel array 730 can activate pixels by the unit of row line based on arow line control of the row driver 733. The readout driver 735 canoutput a signal of each sub pixel level along the column line inaccordance with the control of the timing signal generator 740.

The unit pixels 810 to 840 of the pixel array 730 can each include amicro lens and/or each color filter installed at an upper part. Themicro lens can increase each light gathering power. Each color filtercan transmit or block light of a specific spectrum range. The sub pixelswithin the unit pixel can include a photo diode. For example, the subpixels can share one micro lens and color filter, and can use respectivephoto diodes. For example, in case where one unit pixel includes foursub pixels, the four sub pixels can convert an optical signal receivedthrough one micro lens and color filters, into an electrical signalthrough respective corresponding photo diodes.

The row driver 733 can drive control signals for controlling respectiveoperations of a majority of sub pixels, into the pixel array 730 underthe control of the timing signal generator 740. For example, the controlsignals can include a signal for selecting the sub pixels and a signalfor resetting. The readout driver 735 can include elements (for example,Analog to Digital Converter (ADC), etc.) for selecting the column linesof the pixel array 730 and reading-out signals of sub pixel levels ofthe selected column lines.

According to an example embodiment, the readout driver 735 can outputsignals of respective sub pixel levels of the unit pixels 810 to 840.For example, the readout drivers 810 to 840 can read-out image signalssensed in the respective sub pixels. The processor 400 can treat and/orprocess the sub pixel signals and generate angular information and/ordistance information (e.g., depth data).

FIGS. 9A, 9B and 9C are diagrams illustrating example structures of anexample unit pixel and sub pixels of an example image sensor accordingto various example embodiments of the present disclosure.

FIG. 9A is a diagram illustrating an example in which the unit pixel ofthe image sensor includes two sub pixels. FIG. 9B is a diagramillustrating an example in which the unit pixel of the image sensorincludes four sub pixels. FIG. 9C is a diagram illustrating an exampleof a structure of the unit pixel having the two sub pixels as in FIG.9A.

Referring to FIG. 9A, the unit pixel of the image sensor 700 can includeat least one micro lens 920, at least one color filter 930, and/or alight receiving part 940 including two photo diodes (PD1 and PD2). Forexample, the unit pixel of FIG. 9A can generate two sub pixel signals.The following description will be made in which the image sensor havingthe unit pixel structure of FIG. 9A is used interchangeably with a 2-PDimage sensor. Referring to FIG. 9B, the unit pixel of the image sensor700 can include one micro lens 920, one color filter 930, and/or a lightreceiving part 950 including four photo diodes (PD1 to PD4). Forexample, the unit pixel of FIG. 9B can generate four sub pixel signals.The following description will be made in which the image sensor havingthe unit pixel structure of FIG. 9B is used interchangeably with a 4-PDimage sensor.

In FIG. 9A or FIG. 9B, the light receiving part 940 or 950 can convertincident light received through the lens 910 into an electrical signal,and generate the converted electrical signal. The light receiving part940 or 950 can include doping regions of a laminated form. The dopingregions can be laminated based on an incident angle of the incidentlight.

Referring to FIG. 9C, the image sensor 700 can include a plurality ofunit pixels. Each unit pixel 960 can include at least two or more photodiodes. And, a barrier 970 can be located between the respective photodiodes. For example, at least one color filter 930 can be located on theplurality of photo diodes. Also, for example, at least one micro lens920 can be located on the plurality of photo diodes. The micro lens 920can be, for example, located on the color filter 930. For example, lightincident on each unit pixel 960 can be incident on the respectivedifferent photo diodes via at least one micro lens 920 and at least onecolor filter 930. The image sensor 700 can output information for focusdetection in accordance with a phase difference of light that isincident on the respective photo diodes.

According to an example embodiment, a lens 910 can be operativelycoupled with an actuator for Optical Image Stabilization (OIS) or AutoFocus (AF).

According to an example embodiment, the color filter 930 can be a Red(R) filter, a Green (G) filter, or a Blue (B) filter, or can be a yellowfilter, a magenta filter, or a cyan filter, or the like, but is notlimited thereto.

According to an example embodiment, the color filter 930 can be formedon the photo diode 940 or 950 based on an incident angle of incidentlight, and can have a Bayer pattern. The Bayer pattern can arrangefilters for accepting the brightness of each of red, green, and/or blueon a two-dimensional plane in order to gather the brightness and colorof a target and make image data made with points. Each unit pixelforming a lattice network under a Bayer pattern color filter recognizesnot the total natural color but only assigned color among red, green,and/or blue and, by interpolating this, analogizes the natural color.

According to an example embodiment, the micro lens 920 can be formed tocorrespond to the photo diode 940 or 950 on the color filter 930 asmaintaining a tiling angle at which the color filter 930 is laminated.The OIS lens 910 can be, for example, located inside a lens mount (notshown), and can gather light.

According to an example embodiment, the processor 400 can process (forexample, add and average) signals of respective sub pixels of a unitpixel, and output the processed signals as image data. According to anexample embodiment, the processor 400 can determine a phase differencebetween the signals of the sub pixels included in the unit pixel, togenerate distance information (e.g., depth information). For example, inFIG. 9A, the processor 400 can determine a phase difference between twosub pixel signals of the unit pixel, to generate depth information. Theprocessor 400 can process the two sub pixel signals to generate colorimage data. For example, if the image sensor 700 having the pixel array730 of a High Definition (HD) (1,280×720) size has a 4-PD structure inwhich one unit pixel has four sub pixels, the image sensor 700 can reada color value of each photo diode and output image data having aresolution of Quad HD (QHD) (2,560×1,440).

FIG. 10A and FIG. 10B are diagrams illustrating examples of a unit pixeland sub pixels of an image sensor according to various exampleembodiments of the present disclosure. FIG. 10A is an example diagram inwhich the unit pixel of the image sensor includes two sub pixels. FIG.10B is an example diagram in which the unit pixel of the image sensorincludes four sub pixels.

The image sensor 700 can include a plurality of unit pixels. Each unitpixel can include a plurality of sub pixels. The R pixel can be a pixelimage for red color, and the G pixel can be a pixel image for greencolor, and the B pixel can be a pixel image for blue color.

According to an example embodiment, one unit pixel can include at leasttwo or more sub pixels (e.g., photo diodes). Each unit pixel can includeone micro lens and color filter. The unit pixel can output a signal ofone sub pixel level and thus one unit pixel can include a plurality ofdata.

According to an example embodiment, the unit pixel can sum up signals ofat least two or more sub pixel levels included and output the signals asone image data as well. The unit pixel can output phase differenceinformation for determining a phase difference of light that areincident on at least two sub pixels as well. For example, in case wherethe unit pixel includes the two sub pixels as in FIG. 10A, the phasedifference information can include information for determining a phasedifference between left/right pixels. For example, in case where theunit pixel includes four sub pixels as in FIG. 10B, the phase differenceinformation can include information for determining a phase differencebetween the up and down sub pixels or information for determining aphase difference between the left and right sub pixels. For example, thephase difference information can include information for determining aphase difference between diagonal sub pixels that are locateddiagonally. For example, the unit pixel can output information fordetermining a phase difference between sub pixels of a specific color aswell. For example, the unit pixel can output information for determiningcolor information and a phase difference only from the G pixel thatrelatively much accepts light among the RGB pixels, and output onlycolor information from the remnant R pixel and B pixel as well.

FIG. 10A and FIG. 10B illustrate the Bayer pattern based on red, green,and blue, but an example embodiment of the present disclosure is notlimited to this and can use various filter patterns.

According to various example embodiments, the processor 400 can acquiredistance information and/or image data, based on sub pixel signalsoutput from the image sensor 700. For example, the processor 400 canoperate based on a photographing mode and condition and acquire thedistance information (e.g., depth information) and/or the image data.

In an example embodiment, the processor 400 can determine a phasedifference based on respective sub pixel signals of the image sensor700. The processor 400 can extract distance information by using phasedifference information.

In an example embodiment, the processor 400 can generate image databased on respective sub pixel signals of the image sensor 700. Forexample, in case where the image sensor 700 is an image sensor having a2-PD structure, the processor 400 can process (for example, add oraverage) a 1st sub pixel signal and a 2nd sub pixel signal of each unitpixel, to generate image data of the unit pixel.

In an example embodiment, the processor 400 can generate phasedifference information and image data based on respective sub pixelsignals of the image sensor 700. For example, in case where the imagesensor 700 is an image sensor having a 2-PD structure, the processor 400can determine a phase difference between a 1st sub pixel signal and a2nd sub pixel signal of each unit pixel, to generate distanceinformation. The processor 400 can process the 1st sub pixel signal andthe 2nd sub pixel signal, to generate image data.

In an example embodiment, the processor 400 can generate phasedifference information based on respective sub pixel signals of theimage sensor 700. In a case where the image sensor 700 is an imagesensor having a 2-PD structure, the processor 400 can selectivelyprocess a 1st sub pixel signal and/or a 2nd sub pixel signals. Forexample, the processor 400 can differently adjust an operation ofprocessing image data based on an attribute (e.g., color) of a unitpixel. For example, in case where the attribute of the unit pixel is redor blue, the processor 400 can control the image sensor 700 to outputimage data summing up signals (e.g., 1st signal and 2nd signal) ofrespective sub pixel levels. Also, in case where the attribute of theunit pixel is green, the processor 400 can control the image sensor 700to process data (e.g., 1st signal and 2nd signal) of respective subpixel levels into image data of a unit pixel level and output the imagedata.

FIG. 11A and FIG. 11B are diagrams illustrating an example operation ofan image sensor including a top surface phase difference sensor. FIG.11A is a diagram illustrating positions of objects according to variousexample embodiments of the present disclosure.

In a state where objects 1110 to 1130 are arranged in front of theimaging device 1100 as in FIG. 11A, a user can focus the imaging device1100 on a desired object and photograph the desired object. The imagingdevice 1100 can include the image sensor 700 having the construction ofFIG. 7. FIG. 11A illustrates an example in which the object 1110 islocated closest to a position of the imaging device 1100 at a distance(D1), the object 1130 is located farthest from the imaging device 1100at a distance (D3), and the object 1120 is located at an intermediatedistance from the imaging device 1100 at a distance (D2). In a state inwhich the objects are arranged in front of the imaging device 1100 as inFIG. 11A, if the user focuses the imaging device 1100 on a specificobject and photographs the specific object, a photographed image can beacquired as an image in which the focused object becomes clear and otherobjects become less clear based on a distance spaced apart from thefocused object.

Phase difference detection pixels of the imaging device 1100 can outputrespectively different phase difference values for the objects 1110 to1130. For example, in case where the imaging device 1100 is focused onthe object 1120, the object 1120 can be an object of a 1st focus region,and the object 1110 can be an object of a 2nd focus region, and theobject 1130 can be an object of a 3rd focus region.

FIG. 11B is a diagram illustrating an example image sensor according tovarious example embodiments of the present disclosure.

Referring to FIG. 11B, the imaging device can include at least one phasedifference detection pixel 1150 (e.g., top surface phase differencesensor) and at least one image pixel 1160 (e.g., color pixel). In FIG.11B, a 1st image 1170 can be an image photographing a 1st object (e.g.,object 1110 of FIG. 11A). A 2nd image 1180 can be an image photographinga 2nd object (for example, object 1120 of FIG. 11A). A 3rd image 1190can be an image photographing a 3rd object (for example, object 1130 ofFIG. 11A). According to an example embodiment, the electronic device(for example, processor 400 of FIG. 4) can acquire relative distanceinformation (e.g., depth information) between the objects 1110 to 1130sensed by the image sensor, based on a phase difference value sensed inthe at least one phase difference detection pixel 1150. For example, theelectronic device can acquire a distance to the 1st object 1110, basedon a value of at least one phase difference detection pixel included inthe 1st image 1170. For example, in case where the value of the phasedifference detection pixel corresponding to the 1st image 1170 is avalue corresponding to the 1st focus region, the electronic device candetermine that the imaging device is focused on the 1st image 1170, andcan acquire a distance (for example, D1 of FIG. 11A) to the 1st object1110 corresponding to the 1st image 1170 based on a lens rangefinder.The electronic device can divide the image sensor into sub regions (orplurality of regions) R₁₁ to R_(mn) (e.g., row lines R₁ to R_(m) andcolumn lines R₁ to R_(n)), and acquire a distance between objectscorresponding to the respective sub regions based on values of phasedifference detection pixels included in the respective sub regions. Forexample, in case where the value of the phase difference detection pixelcorresponding to the 1st image 1170 is the value corresponding to the1st focus region, and a value of a phase difference detection pixelcorresponding to the 2nd image 1180 is a value corresponding to the 2ndfocus region, the electronic device can acquire a relative distancebetween the 1st image 1170 and the 2nd image 1180.

FIG. 12A and FIG. 12B are diagrams extending and illustrating a sectionof one portion of an example image sensor according to various exampleembodiments of the present disclosure.

Referring to FIG. 12A, pixels 1251 and 1255 of the image sensor (forexample, pixel array 730 of FIG. 7) can include wiring regions 1213,light receiving regions 1215, and imaging lenses, for example, microlenses 1217.

The wiring regions 1213 can be arranged to be spaced a constant distanceapart on one surface of a substrate 1211 of the image sensor. The lightreceiving regions 1215 can be arranged between the wiring regions 1213,respectively. For instance, the wiring regions 1213 and the lightreceiving regions 1215 can be arranged mutually alternately on onesurface of the substrate 1211. Elements such as a power source or signalline and a transistor can be arranged in the wiring regions 1213. Thelight receiving regions 1215 are photoelectric transformation regionssubstantially detecting and converting light and/or an image into anelectrical signal. The light receiving regions 1215 can forward theelectrical signal based on information of the detected image, to thewiring region 1213.

The respective unit pixels 1251 and 1255 can include one light receivingregion 1215, a wiring and transistor of the wiring region 1213 coupledto the light receiving region 1215, and/or the micro lens 1217 arrangedcorrespondingly to the corresponding light receiving region 1215. Amongthe unit pixels 1251 and 1255, a partial pixel (e.g., 1251) can operateas an image detection pixel (e.g., color pixel). Among the unit pixels1251 and 1255, a partial pixel (e.g., 1255) can be, for example,constructed as a phase difference detection pixel having a phaseseparation structure.

The phase difference detection pixel 1255 can include a phase separationstructure installing a light blocking film 1219 on the light receivingregion 1215. Each micro lens 1217 can be arranged in a state where anoptical axis (P) is coincident with the light receiving region 1215corresponding to the micro lens 1217. A cavity 1229 is provided betweenthe wiring regions 1213 upward the light receiving region 1215. Thelight blocking film 1219 can be arranged in the cavity 1229 between thewiring regions 1213. The light blocking film 1219 can cover a part (forexample, about half) of a surface area of the light receiving region1215 at one side of the light receiving region 1215, and partially blocklight introduced into the light receiving region 1215.

Referring to FIG. 12B, the phase difference detection pixel 1255 can bepaired as denoted by reference numerals 1255 a and 1255 b of FIG. 12Band be arranged to be mutually adjacent or be partially spaced apart. Alight blocking film 1219 a can be arranged in a 1st phase differencedetection pixel 1255 a among the paired phase difference detectionpixels 1255. The light blocking film 1219 a can be arranged in aposition that is not overlapped, for example, is offset with respect toa light blocking film 1219 b arranged in a 2nd phase differencedetection pixel 1255 b. If the 1st phase difference detection pixel 1255a detects light that passes the one side of the micro lens 1217, the 2ndphase difference detection pixel 1255 b can detect light that passes theother side of the micro lens 1217. An image sensor and/or an electronicdevice such as a camera or portable terminal equipped with the imagesensor can compare values detected by the respective phase differencedetection pixels 1255 and measure a focus adjustment state of the imagesensor.

According to an example embodiment, the phase difference detection pixel1255 can output a different value based on a 1st focus state (e.g.,solid line 1261) in which a focus of an object is positioned at theimage sensor, a 2nd focus state (e.g., dotted line 1263) in which thefocus of the object is positioned between the image sensor and theobject, and a 3rd focus state (e.g., chain double-dashed line 1265) inwhich the focus of the object is positioned in a region other than theimage sensor and between the image sensor and the object. The electronicdevice can identify a relative distance between objects, based on the1st focus state to the 3rd focus state.

In FIG. 12A and FIG. 12B, a description according to an exampleembodiment has been made for, for example, the phase differencedetection pixel having the light blocking film, in order to explain thestructure and operation of the phase difference detection pixel. But itwill be understood that the phase difference detection pixel accordingto various example embodiments can include phase difference detectionpixels having phase difference separation structures of various forms(or structures).

As above, when a user photographs an image with the imaging device 1100,a focus can be made in a set object position. The phase differencedetection pixel (e.g., top surface phase difference sensor) 1150arranged in the image sensor can extract distance information of afocused object and non-focused other objects. The electronic device canget a phase difference value by each pair of the phase differencedetection pixels (or top surface phase difference sensors) 1150. Theelectronic device can also divide a region of the image sensor into subregions as illustrated in FIG. 11B, and get phase difference values,using phase difference detection pixels (or top surface phase differencesensors) corresponding to the respective sub region.

As illustrated in FIG. 11B, the following description will be made byway of an example in which distance information of an object isdetermined using phase difference detection pixels corresponding torespective sub regions. An image can include two or more objects. Whenthe electronic device determines distance information of these objects,the electronic device can get a distance between a focused object andnon-focused objects (e.g., objects focused in front or focused in rear).

According to an example embodiment, the electronic device can apply animage processing technique adapted (or suitable) to a photographingobject, to an acquired image, to generate a new image adapted to auser's preference (or desired by a user). The image processing techniqueapplied to the acquired image can be automatically determined and chosenby the electronic device, or the electronic device can display analyzedimage processing techniques such that the user can choose.

According to an example embodiment, the electronic device can analyzeobject information that is included in an image acquired in the imagesensor, and determine the image processing technique, and can apply thedetermined image processing technique to the acquired image and displaythe image. For example, the image processing technique can be a methodof determining an image filter applicable to the acquired image. Forexample, the acquired image can be a food image.

According to an example embodiment, the electronic device can acquireimage data associated with food, and extract distance informationbetween food and the electronic device based on the food image data, andgenerate configuration information (e.g., size of a food image, positionthereof, distance information of food, etc.) of food based on theextracted distance information, and determine an image filter based onthe configuration information of the food. As illustrated in FIG. 6A toFIG. 6E, the image filter can be a selection of one or more of thelinear filter, the circular filter, the lighting filter, the macrofilter, and/or the selective focus filter.

FIG. 13A and FIG. 13B are diagrams illustrating an example of displayingan acquired image and distance information that is extracted based onacquired image data in an electronic device according to various exampleembodiments of the present disclosure.

According to an example embodiment, the electronic device can include afood mode. If a user triggers a food photographing command, theelectronic device (for example, photographing setting module 520) canapply an image processing technique to image data output from the cameramodule 510, to generate a new image.

The electronic device can acquire an image of FIG. 13A. The image ofFIG. 13A is a food image, and can have a structure in which food iscontained in a container. For example, objects included in the image canbe a container, food, etc. The image sensor of the camera module 510 canread out the image of FIG. 13A. The processor 400 can extractinformation such as a position relationship between the objects (forexample, container and/or food) included in the image output from thecamera module 510, a position relationship between the object and aphotographing unit (e.g., camera), a posture of the photographing unit,etc. For example, if a unit pixel is an image sensor (for example, 2-PDimage sensor, 4-PD image sensor, etc.) including a plurality of subpixels as in FIG. 8, the processor 400 can acquire (estimate) distanceinformation based on a phase difference between sub pixel signals.

The processor 400 can estimate configuration information (e.g., depthmap) that can figure out the position relationship between theelectronic device and the objects from the estimated distanceinformation. A method of estimating the distance information can use atop surface phase difference image sensor, a focus-adjustment imagesensor, an IR or ultrasonic image sensor, etc. besides a multi PD imagesensor. The distance information (e.g., depth map) of the image can beexpressed in the form of FIG. 13B as well.

FIGS. 14A, 14B, 14C and 14D are diagrams illustrating an example ofimage processing of an electronic device according to various exampleembodiments of the present disclosure.

The processor 400 can sense a posture of the electronic device and achange of the posture, based on an output of the sensor module 430. Theprocessor 400 can determine a feature (e.g., constituent element,position relationship, etc.) of an object, based on image data outputfrom an image sensor and estimated configuration information. Theprocessor 400 can determine an image filter based on the determinedfeature of the object. For example, if the feature of the object has aform in which it is long in a horizontal, vertical or diagonaldirection, the electronic device can determine a linear filter. Forexample, if a feature of an object 1450 has a circular or oval (forexample, food contained in a spherical container) form as illustrated inFIG. 14C, the electronic device can determine a circular filter. Afterdetermining the image filter, the electronic device can apply thedetermined image filter to an image acquired from the image sensor, togenerate a new image.

For example, in FIG. 14A, image data can include a main object image1410 and a background image 1420. If the linear filter is determinedbased on the image of FIG. 14A, the electronic device can apply thelinear filter to the acquired image of FIG. 14A, to generate a new imageof FIG. 14B. The generated new image may include a clearly displayedregion (1430) and a blur-processed region (1440). In FIG. 14C, imagedata can include a main object image 1450 and a background image 1460.If the circular filter is determined based on the image of FIG. 14C, theelectronic device can apply the circular filter to the acquired image ofFIG. 14C, to generate a new image of FIG. 14D. The generated new imagemay include a clearly displayed region (1470) and a blur-processedregion (1480).

The center of a blur in the image to which the image filter is appliedcan be a focused point that is a criterion. A focus can be the center ofthe image or can be diversified based on a touch, etc. An imagefiltering method can perform blur processing around a focus positionwithout performing the blur processing in the focus position.

FIG. 15 is a block diagram illustrating an example image processingdevice according to various example embodiments of the presentdisclosure.

Referring to FIG. 15, the electronic device can set an image processingtechnique suitable to an acquired image, and can apply the set imageprocessing technique to the acquired image, to generate a new image thata user desires. According to various example embodiments of the presentdisclosure, the electronic device can analyze the acquired image and setautomatically the image processing technique. The image processingtechnique set in the electronic device can be used for selecting theimage desired by the user. According to an example embodiment, theelectronic device can set the image processing technique by analyzingthe acquired image and a photographing environment (for example, aphotographing condition such as a focus, light exposure, white balance,etc., a photographing posture, etc. dependent on a photographing angleof the electronic device, a motion, etc.).

The electronic device can set a camera module based on the setphotographing condition, and acquire an image through the camera module.The photographing condition can include a photographing mode that usesthe image processing technique according to various example embodimentsof the present disclosure. The camera module can include an image sensor1510. The image sensor 1510 can sense an object based on thephotographing condition and convert the sensing result into anelectrical image signal. Image data can include brightness information,color information, a signal for extracting distance information betweenthe object and the electronic device, etc.

If the image processing technique is determined, the electronic devicecan drive the camera module and perform a photographing operation. Anoperation of the camera module can be an operation of displaying apreview image. The operation of displaying the preview image can be anoperation of processing image data acquired in the image sensor 1510 anddisplaying the image data in a display module 1560.

The image sensor 1510 can be an image sensor (for example, 2-PD imagesensor or 4-PD image sensor) having a structure of FIG. 8. The imagesensor 1510 can be an image sensor that includes a pixel sensor and topsurface phase difference sensors as in FIG. 11B. The image sensor 1510can include a TOF image sensor, a structured-light image sensor, and/oran IR or ultrasonic measurement device.

According to an example embodiment, the image sensor 1510 can be aone-piece image sensor that implements as one device a constructionoutputting an image and a signal for extracting distance information.According to an example embodiment, the image sensor 1510 can be astereo-structured image sensor in which each of an image sensorgenerating image data and a sensor (for example, depth sensor)generating a signal for extracting distance information are constructedindependently.

According to an example embodiment, the image sensor 1510 can generateimage data and signals for extracting distance information, at the sametime. According to an example embodiment, the image sensor 1510 cangenerate the image data and the signals for extracting distanceinformation, at different time.

According to an example embodiment, the image sensor 1510 can generateand map image data and signals for extracting distance information bypixel or by region. According to an example embodiment, the image sensor1510 can generate and not to map the image data and the signals forextracting the distance information by pixel or by region.

An image processing module 1520 may include various image processingcircuitry and program elements that can extract image data and distanceinformation by processing an output from the image sensor 1510. Theimage processing module 1520 can include an image processing unit (e.g.,Image Signal Processor (ISP)) and an information acquisition unit. Theimage processing unit can extract and process image data from a 3D imagethat is output in the image sensor 1510. The information acquisitionunit can extract distance information between an object and theelectronic device, by extracting signals for extracting the distanceinformation, from the 3D image output in the image sensor 1510.

The image processing unit can perform an image pre-processing operationand an image post-processing operation. First, the image pre-processingoperation can perform Auto White Balance (AWB), Auto Exposure (AE), AutoFocusing (AF) extracting and processing, lens shading correction, deadpixel correction, knee correction, etc., for images. The imagepost-processing operation can include color interpolation, ImageProcessing Chain (IPC), color conversion, etc. The color interpolationoperation can perform a function of converting (e.g., full-colorconvert) pixels output in the image sensor 1510 into color includingthree colors, RGB. The IPC can perform noise reduction ofcolor-interpolated images, gamma correction, luminance correction, etc.And, the color conversion operation can convert a corrected image (forexample, raw data, Bayer data) into a YUV (or YCbCr) image.

The information acquisition unit can determine distance informationbetween an object and the electronic device, based on pixel signalsoutput in the image sensor 1510. For example, in case where the imagesensor 1510 is an image sensor including a top surface phase differencesensor, the information acquisition unit can extract distanceinformation between the object and the electronic device, based onsignals of the top surface phase difference sensor. The informationacquisition unit can generate photographing environment information ofthe electronic device. The photographing environment information caninclude photographing meta data (for example, exchangeable image fileformat (exif)), and/or photographing posture information of theelectronic device. The distance information estimated in the informationacquisition unit can be a depth map. The depth map information can bemapped with pixels of image data.

For example, the image sensor can be a 2-PD image sensor. The imageprocessing unit can sum up or average sub pixel signals output in the2-PD image sensor 1510, to generate a signal of a unit pixel.Thereafter, the image processing unit can pre-process and post-processunit pixel signals to generate image data. By determining a phasedifference between a 1st sub pixel signal and 2nd sub pixel signal ofthe unit pixel, the information acquisition unit can estimate distanceinformation. The phase difference between the 1st and 2nd sub pixelsignals determined in the information acquisition unit can be used asdistance information between the electronic device and the object.

According to an example embodiment, an image sensor of the camera module510 can be a two-dimensional image sensor generating image data. If theimage sensor is the two-dimensional image sensor, the image processingmodule 1520 can extract a contour line of image data and generateinformation similar with distance information. For example, if the imagesensor is the two-dimensional image sensor, the information acquisitionunit can extract a contour line of image data generated in the imagesensor and generate information similar with distance information. Thecontour line of the image data can be extracted based on a change ofcolor data, a brightness of the image, etc.

The recognition module 1530 may include various processing circuitry andprogram elements that can recognize configuration information (forexample, a range of an object, a position of the object within an image,etc. based on distance information) of the object, based on image dataand distance information (or contour line information of the image data)that are generated in the image processing module 1520. For example, therecognition module 1530 can recognize at least one object includedwithin the image, based on distance information between the object andthe electronic device output in the image processing module 1520.According to an example embodiment, the recognition module 1530 canestimate the configuration information of the object based on thedistance information. According to an example embodiment, therecognition module 1530 can estimate the configuration information ofthe object based on the image data and the distance information.

Configuration information of an object that is recognized in therecognition module 1530 can be estimated based on a depth map. Theconfiguration information of the object can include information such asa form of the object (for example, linear shape, circular shape, ovalshape, etc.), depths of the objects, a size of the object, a position ofthe object, etc. The filter determination module 1540 may includevarious circuitry and program elements that can determine an imagefilter based on the configuration information of the object. The imagefiler can be filters of various forms as illustrated in FIG. 6A to FIG.6E. The filter determination module 1540 can select one or a pluralityof image filters based on the configuration information of the object.

The filter applying module 1550 may include various circuitry andprogram elements that can apply an image filter determined in the filterdetermination module 1540, to an image-processed acquisition image, togenerate a new image. The display module 1560 can display the new imagegenerated in the filter applying module 1560, as a preview image.

During the duration of analyzing a depth map and an image anddetermining configuration information (for example, object region) of anobject, the electronic device can provide a notification of notifyingthat it is in course of image processing to the display module 1560. Forexample, the electronic device can apply an image processing effect(e.g., filter effect) which will be applied to a final photographingimage or is similar with this, to a preview screen, and show the previewscreen. By doing so, the electronic device can notify that it is now inprocessing, and concurrently display a process of generating the finalresult. For example, when applying an image filter to an acquired image,the electronic device can change a size of a preview image or change anattribute (for example, blur intensity and exposure value) of thepreview image, thereby getting a user to feel the feeling of luxurywithout being bored.

According to an example embodiment, an image acquired in the imagesensor 1510 can be a food image. For example, if food is included amongobjects, the image sensor 1510 can generate image data. The image datacan include the food image and/or a signal for extracting distanceinformation of food. The image processing module 1520 can process thefood image data, and can generate distance information (e.g., depth map)based on the food image data. The food image can be expressed by a depthmap of an object and posture information of the electronic device. Forexample, the depth map can be sensed and acquired by a 2-PD imagesensor, or can be acquired by a top surface phase difference imagesensor or a focus adjustment image sensor as well. The depth map can beobtained through a separate IR or ultrasonic measurement device as well.The posture information of the electronic device can be obtained througha sensor (for example, acceleration sensor and/or gyro sensor) installedin the electronic device.

The recognition module 1530 can recognize configuration information of afood image (e.g., a range (or size) of the food image, a position, adistance, etc.), based on depth map information or the food image andthe depth map information. For example, the recognition module 1530 canrecognize configuration information of food that is an object, based onacquired distance information that is capable of further including adepth map and/or posture information of the electronic device. Thefilter determination module 1540 can determine the most suitable imagefilter that will be applied to an object, based on the configurationinformation of the object (for example, food). The determined imagefilter can be one of a plurality of previously prepared image filters ora combination of two or more image filters. For example, the recognitionmodule 1530 can extract a range of food that is an object and a positionthereof, using the distance information (for example, depth map). Thefilter applying module 1540 can determine the most suitable image filterbased on the extracted food range and position. For example, if therange of the food recognized in the recognition module 1530 is in a formin which it is long horizontally, the filter determination module 1540can determine the linear filter of FIG. 14B. If the range of the foodrecognized in the recognition module 1530 is of a circular shape, thefilter determination module 1540 can determine the circular filter ofFIG. 14D.

The filter applying module 1550 can apply an image filter determined inthe filter determination module 1540, to image data output in therecognition module 1530, to generate a new image. For example, thefilter applying module 1550 can clearly display an image (for example, aregion of a main object recognized in the recognition module 1530) of aregion to which the image filter is applied, and can give a blur effectto a peripheral region other than this. The region to which the imagefilter is applied can be set based on an object constituent element thatis recognized by distance information. For example, the center of a blurin the image filter can be a focused point that is a criterion. A focuscan be the center of the image filter, or can be diversely set based ona touch, etc. According to an example embodiment, an image filteringmethod can perform blur processing around a focus position withoutperforming the blur processing in the focus position. According to anexample embodiment, the image filtering method can perform the blurprocessing in the focus position and around the focus position, atmutually different blur intensities.

The filter applying module 1550 can apply an effect different from ablur effect, to images of a region to which an image filter is applied.For example, the filter applying module 1550 can apply a lighting effectof giving a bright feeling or a macro effect of extending a specificregion as well. For example, in case where a food range is of a circularshape, the filter determination module 1540 can determine a circularfilter and a lighting filter. If so, the filter applying module 1550 canset a circular filter region to a region in which an object is located,and apply the lighting effect to an object image.

If a new image is generated in the filter applying module 1550, thedisplay module 1560 can display the generated new image. The imagedisplayed in the display module 1560 can be a preview image. Whileextracting distance information (e.g., depth map) and estimating anobject region in order to apply an image filter, the electronic devicecan display that it is in processing through the display module 1560.

According to an example embodiment, while determining an object regionthat will apply an image filter, the electronic device can display onthe display module 1560 a preview image having an effect similar with anew image that will be image-processed and generated. For example, whenthe electronic device displays the preview image, the electronic devicecan change and display a section to which the image filter (for example,linear filter, circular filter, etc.) will be applied, or can change anddisplay a blur intensity and/or a brightness (for example, exposurevalue) of a peripheral image. The section to which the image filter isapplied can be the entire image. The change of the section to which theimage filter is applied can represent a change of a section applyingprocessing (e.g., processing effect) within the image filter.

Accordingly, a user can check a process of dynamically changing anoperation of analyzing food through a preview image and automaticallysetting an image filter effect and an excellent picture quality.According to an example embodiment, when the electronic device displaysthe preview image, the electronic device can display the preview imagebased on posture information of the electronic device. For example, theposture information of the electronic device can be acquired based on asensor module, and can be provided from the recognition module 1530 tothe filter applying module 1550. The filter applying module 1550 canapply an image filter to an acquired image in accordance with aphotographing posture of the electronic device, thereby generating thepreview image. The filter applying module 1550 can output the postureinformation of the electronic device to the display module 1560,together with the preview image. The display module 1560 can display thepreview image that is based on the photographing posture of theelectronic device, and can display associated photographing postureinformation.

In an example embodiment, the electronic device can change anapplication region (e.g., focus position) of an image filter inaccordance with a user's selection, in course of displaying a previewimage. For example, the electronic device can change the image filterand a central position by user's designation (e.g., touch, etc.) in thepreview image. For example, if a filter region is set by a user (forexample, if a user's drawing input is recognized), the recognitionmodule 1530 can generate configuration information by the filter regionthat is set by the user. The filter determination module 1540 can setthe image filter based on the filter region set by the user. In apreview mode, the filter applying module 1550 can apply the image filterset by the user, to generate a new image (for example, preview image).The user can set a focus point of the image filter, by a touch input,etc. If the focus point is set, the filter applying module 1550 canapply the set image filter, centering on the focus point that is set bythe user.

If a user generates a capture command in course of displaying a previewimage, the electronic device can store a new image generated in thefilter applying module 1550, in a memory. According to an exampleembodiment, the electronic device can store all of image data acquiredin the image processing module 1520 and the new image generated in thefilter applying module 1550.

An electronic device according to various example embodiments of thepresent disclosure can include a camera module, a memory module, and aprocessor operatively coupled with the camera module and the memorymodule. The processor can acquire an image through the camera module,extract distance information based on the acquired image, determine animage processing technique for an object based on the extracted distanceinformation, apply the determined image processing technique to theacquired image, and display the applied image.

The camera module can include an image sensor generating image data anda signal for extracting the distance information between the electronicdevice and the object.

The image sensor of the camera module can include an array of unitpixels, and the unit pixel can include two sub pixels. The processor canextract the distance information based on a phase difference betweensignals of the sub pixels of the unit pixel, and average the signals ofthe sub pixels to generate the image data.

The image sensor of the camera module can include a top surface phasedifference sensor and pixel sensors, and the processor can extract thedistance information based on a signal of the top surface phasedifference sensor, and generate the image data based on signals of thepixel sensors.

The processor can extract the distance information based on the image,and recognize the object based on the distance information.

The image processing technique is an image filter, and the processor canextract configuration information of the recognized object, anddetermine the image filter based on the extracted configurationinformation, and apply the determined image filter to the acquired imagedata, to generate a new image.

While determining the image processing technique, the processor candisplay a preview image, and the preview image can be a preview image towhich a dynamic filter being based on a blur, an exposure change, etc.is applied.

The electronic device can further include a sensor module sensing aposture of the electronic device, and the processor can display postureinformation of the electronic device in the new image that is displayedas the preview image.

The processor can select at least one filter among a linear filter, acircular filter, a lighting filter, a macro filter, and a selectivefocus filter in the configuration information, and apply the selectedfilter to the image data.

The generated new image can be an image that is blur-processed in itsperipheral region with a criterion of a focus of the image filter. Thatis, the generated new image can be an image that is blur-processed inits peripheral region other than a focus position of the image filter.

The image filter can be an image filter applied to a food image.

At capture request, the processor can store the new image in the memorymodule.

The camera module can include an image sensor generating two-dimensionalimage data. The processor can extract contour line information of theobject based on the image data, and recognize configuration informationof the object based on the contour line information, and determine animage filter based on the configuration information, and apply thedetermined image filter to the image data.

FIG. 16 is a flowchart illustrating an example method of processing animage in an electronic device according to various example embodimentsof the present disclosure.

Referring to FIG. 16, in operation 1611, if a photographing command isgenerated, the electronic device (for example, processor 400 of FIG. 4)can set a photographing condition, and control a camera module toinitiate photographing. The photographing mode can be a photographingmode of applying an image filter. If the photographing is initiated, animage sensor of the electronic device can convert an optical signalincident through a lens, into image data. The image data can includebrightness information and color information of an image, and caninclude a signal for extracting distance information of an object.

In an example embodiment, the electronic device can acquire the imagedata and the signal for extracting the distance information, at the sametime. For example, the image sensor can be a one-piece image sensor inwhich a sensor generating the image data and sensors generating a signalfor extracting the distance information are integrated in one circuit.The electronic device can acquire the image data and the signal forextracting the distance information, at the same frame duration in theone-piece image sensor.

In an example embodiment, the electronic device can acquire the imagedata and the signal for extracting the distance information, at mutuallydifferent time. For example, the image sensor can be a stereo imagesensor in which a sensor generating the image data and sensorsgenerating the signal for extracting the distance information areconstructed independently. The electronic device can individuallyacquire the image data and the signal for extracting the distanceinformation in the stereo image sensor.

In an example embodiment, the electronic device may not map the imagedata and the signal for extracting the distance information. Forexample, in the image sensor, the sensor generating the image data andthe sensor for extracting the distance information can be different insize (e.g., resolution). If the sensors are mutually different in size,the electronic device can map the image data and the signals forextracting the distance information, by the unit of pixel. Theelectronic device can map, by region, the image data and the signals forextracting the distance information. For example, the electronic devicecan split the sensor into segments and can map, by segment, the imagedata and the signals for extracting the distance information.

If the electronic device acquires the image data 1613, in operation1615, the electronic device can extract the distance information of theobject based on the three-dimensional image data. The distanceinformation (e.g. depth map) of the object can be distance informationbetween the object and the electronic device.

The image sensor can be an image sensor that cannot generate the signalfor extracting the distance information. If so, in operation 1615, theelectronic device can extract contour line information of the object.For example, in operation 1615, the electronic device can extract acontour line of the image data generated in the image sensor, andgenerate information similar with the distance information. The contourline of the image data can be extracted based on a contour line of animage, a change of color data, a brightness of the image, etc.

After extracting the distance information of the object, in operation1617, the electronic device can recognize configuration information ofthe object based on the distance information of the object and/or theimage data. For example, the configuration information of the object caninclude information such as a range (or size) of the object, a positionwithin an image range, a depth of the object, etc. According to anexample embodiment, the image data output in the image sensor and thesignal for extracting the distance information of the object can begenerated at mutually different time. If a synchronization of the timefor generating a frame of the image data and a frame of the signal forextracting the object distance information is not made, in operation1617, after the electronic device synchronizes the frame of the imagedata and the frame of the signal for extracting the object distanceinformation, the electronic device can recognize configurationinformation of the object based on the image data and the signal forextracting the object distance information.

According to an example embodiment, the image data output in the imagesensor and the signals for extracting the distance information of theobject can be signals whose amplitudes (or resolutions) are not mappedwith each other. If the two signals are not mapped by pixel or byregion, in operation 1617, the electronic device can map the image dataand the signals for extracting the object distance information, andrecognize the configuration information of the object based on themapped two signals. For example, if the resolution (or sensor size) ofthe image data is greater than the resolution of the signals forextracting the distance information, the electronic device can split asize of the image into segments, and map the signals for extracting thecorresponding distance information to the image segments, and recognizethe configuration information of the object based on the mapped twosignals.

In operation 1619, the electronic device can determine an imageprocessing technique based on the recognized configuration informationof the object. The image processing technique can be an image filter. Ifthe image filter is determined, in operation 1621, the electronic devicecan apply the determined image filter to the acquired image data, togenerate a new image. In operation 1623, the electronic device candisplay the generated new image, e.g., a preview image on a display. Theimage displayed on the display can be a preview image.

The preview image can be an image in which an automatically set imagefilter is applied to the acquired image. If a photographing compositionand focus are changed in accordance with a photographing posture (e.g.,photographing angle and/or distance) of a user, in operation 1613, theelectronic device can acquire an image changed by the photographingposture. If the image is changed, the electronic device can extractdistance information based on changed image data. The electronic devicecan recognize configuration information of the newly recognized objectbased on the extracted distance information, and can change the imagefilter based on the recognized configuration information.

A user can check the preview image displayed on the display. While theelectronic device recognizes the object configuration information andsets the image filter based on the generated image, the electronicdevice can display the preview image of an effect similar with the imageto which the image filter is applied. For example, while the electronicdevice determines the image filter, the electronic device can displaythe preview image in various forms on a preview screen, by using a blur,an exposure change, etc. For example, the electronic device can apply adynamic filter (e.g., blur applying, exposure change, etc.) having thefeeling of change, to the preview screen, thereby visually providing theuser with the image filter suitable to the object through processing.

In the case of displaying the preview image, the user can generate adrawing and/or a touch input. If the drawing input is sensed, theelectronic device can set an image filter based on the drawing input,and display an image to which the set image filter is applied, as thepreview image. If the touch input is sensed, the electronic device canset a focus position of the image filter based on a touch position, anddisplay the preview image to which the set image filter is applied.

If the new image to which the desired composition and image filter areapplied is displayed as the preview image, the user can generate acapture command. If the capture command is generated, in operation 1625,the electronic device can recognize this. In operation 1627, theelectronic device can store the new image to which the image filter sapplied. In operation 1627, the electronic device can store all of thenew image to which the image filer is applied and an image acquired inthe image sensor as well.

According to an example embodiment, the image acquired in the imagesensor of the electronic device can be a food image. A user can set afood mode in a photographing menu. If the food mode is selected, inoperation 1611, the electronic device can recognize the setting of thefood mode, and can perform an operation of applying an image filter andgenerating an image. According to an example embodiment, in case wherethe electronic device uses a 2-PD image sensor, the electronic devicecan generate image data and a signal for extracting distanceinformation. For example, a signal of sub pixels constructing a unitpixel can be the signal for extracting the distance information, and anaverage value (or sum-up) signal of the sub pixels can be the imagedata. The image sensor can convert an optical signal incident through alens, and output the optical signal as image information. In operation1613, the electronic device can acquire food image information. Forexample, an object can be food, and the food can have mutually differentdistances between the electronic device and the food. For example, incase where the food is contained in a container, the container and thefood can have mutually different distances with the electronic device.And, even the food positioned within the container can have mutuallydifferent distance information. In operation 1615, the electronic devicecan extract the distance information from image data of the food. Forexample, if a food image is acquired as in FIG. 13A, the electronicdevice can extract a depth map of the food image as in FIG. 13B. Inoperation 1617, the electronic device can estimate configurationinformation (for example, a food range and/or position, a distance ofthe food image, etc.) of the food that will apply the image filter,based on the depth map (or depth map and image data).

For example, if food is included among objects constructing an image,the electronic device can estimate (or sense) a form of the food,position information, a position relationship between the food and theelectronic device. Distance information extracted from the image can beexpressed by a depth map of the image (for example, a food image withinthe entire image acquired) and posture information of the electronicdevice. For example, the depth map can be obtained through a 2-PD imagesensor, or can be acquired through an image sensor including a topsurface phase difference sensor as well. The depth map can be obtainedthrough a separate IR or ultrasonic measurement device as well. Theposture information of the electronic device can be acquired through asensor installed in the electronic device.

In operation 1619, the electronic device can determine the most suitableimage filter that will be applied to the object, based on the distanceinformation (e.g., depth map and/or posture information of theelectronic device). The image filter can be image filters of variousforms, and can be stored in the memory. In accordance with therecognized configuration information (for example, a range of a foodimage, a position thereof, etc.) of the object, the electronic devicecan set one or more image filters among a previously prepared pluralityof image filters.

According to an example embodiment, if a food image has a linear form,the electronic device can set the linear filter of FIG. 6A. According toan example embodiment, if the food image is contained within a containerhaving a circular or polygonal form, the electronic device can set thecircular filter of FIG. 6B. According to an example embodiment, in casewhere a relative change of a depth is smooth, the electronic device canapply the linear filter. According to an example embodiment, in casewhere the change of the depth is sudden, the electronic device can applythe circular filter with a criterion of a region of the depth. Accordingto an example embodiment, if the food image is dark in brightness, theelectronic device can set the lighting filter of FIG. 6C. According toan example embodiment, if the food image has a small size in the entireimage, the electronic device can set the macro filter of FIG. 6D.According to an example embodiment, in case where the electronic deviceemphasizes the food image, the electronic device can set the selectivefocus filter of FIG. 6E.

According to an example embodiment, if a food image has a linear formand is dark in brightness, the electronic device can set the linearfilter and the lighting filter, together. According to an exampleembodiment, if the food image is contained in a circular container andhas a small size in the entire image, the electronic device can set thecircular filter and the macro filter, together. According to an exampleembodiment, if the food image is contained in an oval container and theelectronic device emphasizes the food image within the oval container,the electronic device can set the circular filter and the selectivefocus filter, together.

The electronic device can estimate a food image range in the entireimage based on distance information, and set an optimal image filterthat will be applied to the estimated food image. After setting theimage filter, in operation 1621, the electronic device can apply the setimage filter to the food image, to generate a new image. For example,the electronic device clearly display a region to which the image filter(for example, linear filter and/or circular filter) is applied, andblur-process peripheral region images other than the food image. Forexample, in case where the circular filter and the lighting filter areset, the electronic device can display to emphasize the food image bygiving a lighting effect to food images of a range to which the circularfilter is set. Images of a range to which the image filter is not setcan be blur-processed. According to an example embodiment, when theelectronic device applies the image filter to generate the new image,the electronic device can apply the lighting effect of giving a brightfeeling or a macro effect of extending a specific region.

In operation 1623, the electronic device can display the new image towhich the image filter is applied, as a preview image. The electronicdevice can display another image before providing the preview image towhich the image filter is applied. For example, during the duration ofanalyzing a depth map and an image and determining a region (or objectregion) that will apply the image filter, the electronic device canprovide a notification of notifying that it is in processing to a user.That is, before displaying the new image to which the image filter isapplied, the electronic device can display a preview screen applying aneffect that will be applied to a final photographing image or an effect(e.g., filter effect) that is similar to this. While the electronicdevice generates the new image applying the image filter, the electronicdevice can notify the user that it is now in image processing andconcurrently show a process of generating a final result, therebygetting the user to feel the feeling of luxury of an image processingfunction without being bored. For example, when the electronic deviceapplies the image filter, the electronic device can change a size (e.g.,blur intensity and/or exposure value) of the food image, and display thefood image. For example, the electronic device can dynamically displayan operation of notifying analyzing the food image and automaticallysetting an image filter effect and an excellent picture quality, on thepreview screen.

A method for operating in an electronic device according to variousexample embodiments of the present disclosure can include the operationsof acquiring an image through an image sensor, extracting distanceinformation from the acquired image, determining an image processingtechnique for an object based on the extracted distance information,applying the determined image processing technique to the acquiredimage, to generate a new image, and displaying the new image as apreview image.

The operation of acquiring the image can include the operations ofacquiring image data, and acquiring a signal for extracting the distanceinformation between the electronic device and the object.

The operation of determining the image processing technique can includethe operations of recognizing configuration information of the objectbased on the depth map, and setting an image filter for processing animage based on the configuration information.

The operation of determining the image processing technique can includethe operations of recognizing configuration information based on acontour line of the image data, and setting an image filter forprocessing an image based on the configuration information.

The operation of setting the image filter can set at least one filteramong a linear filter, a circular filter, a lighting filter, a macrofilter, and a selective focus filter based on the recognizedconfiguration information of the object.

The operation of generating the new image can generate an image that isblur-processed in its peripheral region with a criterion of a focus ofthe image filter.

The operation of determining the image processing technique can furtherinclude the operation of displaying a preview image, and the operationof displaying the preview image can display a preview image to which adynamic filter being based on a blur, an exposure change, etc. isapplied.

According to various example embodiments, an electronic device includingan imaging device can analyze an acquired image to automatically set animage processing technique, and apply the set image processing techniqueto the acquired image to generate a new image. If a photographingoperation (for example, food photographing) applying the imageprocessing technique is requested, the electronic device can extractconfiguration information of a main object from image data that isacquired through the imaging device, and can automatically set an imagefilter that will apply based on the extracted configuration information,and apply the image filter to the acquired image, and display an imagein which the main object is emphasized.

Various example embodiments of the present disclosure have beendescribed with reference to the accompanying drawings. It should beunderstood that the various example embodiments are intended to beillustrative, not limiting. Therefore, it will be apparent to one ofordinary skill in the art that various modifications, variations andalternatives will fall within the true spirit and full scope of thepresent disclosure as defined in the appended claims.

What is claimed is:
 1. An electronic device comprising: a display; acamera module comprising an image sensor; a memory module comprising amemory; and a processor operatively coupled with the display, the cameramodule and the memory module, wherein the processor is configured to:acquire image data through the camera module, extract distanceinformation based on the acquired image data, based on the distanceinformation, determine, within the acquired image data, an object,determine, among a plurality of image processing techniques, an imageprocessing technique for the object based on a shape of the object, andapply the determined image processing technique to the acquired imagedata to generate a new image and control the display to display the newimage.
 2. The device of claim 1, wherein the camera module comprises animage sensor configured to acquire the image data and to generate asignal for extracting the distance information of a distance between theelectronic device and the object.
 3. The device of claim 2, wherein theimage sensor of the camera module comprises an array of unit pixels,each unit pixel comprising two sub pixels, and the processor isconfigured to generate the signal for extracting the distanceinformation based on a phase difference between signals of the subpixels of the unit pixel, and to average the signals of the sub pixelsto generate the image data.
 4. The device of claim 2, wherein the imagesensor of the camera module comprises a top surface phase differencesensor and pixel sensors, and the processor is configured to generatethe signal for extracting the distance information based on a signal ofthe top surface phase difference sensor, and to generate the image databased on signals of the pixel sensors.
 5. The device of claim 2, whereinthe processor is configured to extract the distance information based onthe image data, and to recognize the object based on the distanceinformation.
 6. The device of claim 5, wherein the image processingtechnique comprises an image filter, and the processor is configured toextract configuration information of the recognized object, to determinethe image filter based on the extracted configuration information, toapply the determined image filter to the acquired image data, and togenerate the new image.
 7. The device of claim 1, wherein, the processoris configured to control the display to display a preview image in thecourse of determining the image processing technique, and the previewimage comprises a preview image to which a dynamic filter based on ablur and/or exposure change is applied.
 8. The device of claim 6,wherein the processor is configured to select at least one of: a linearfilter, a circular filter, a lighting filter, a macro filter, and aselective focus filter based on the configuration information, and toapply the selected filter to the image data.
 9. The device of claim 8,wherein the generated new image comprises an image that isblur-processed in its peripheral region with a criterion of a focus ofthe image filter.
 10. The device of claim 6, wherein the image filtercomprises an image filter applied to a food image.
 11. The device ofclaim 6, wherein, the processor is configured to store the new image inthe memory module based on a capture request.
 12. The device of claim 7,further comprising a sensor module configured to sense a posture of theelectronic device, and the processor is configured to control thedisplay to display posture information of the electronic device in thenew image that is displayed as the preview image.
 13. The device ofclaim 1, wherein the camera module comprises an image sensor generatingtwo-dimensional image data, and the processor is configured to extractcontour line information of the object based on the image data, torecognize configuration information of the object based on the contourline information, to determine an image filter based on theconfiguration information, and to apply the determined image filter tothe image data.
 14. A method for operating in an electronic device, themethod comprising: acquiring image data through an image sensor;extracting distance information from the acquired image data; based onthe distance information, determining, within the acquired image data,an object, determining, among a plurality of image processingtechniques, an image processing technique for the object based on ashape of the object; applying the determined image processing techniqueto the acquired image data, to generate a new image; and displaying thenew image.
 15. The method of claim 14, wherein the acquiring the imagedata comprises: acquiring the image data; and acquiring a signal forextracting the distance information of a distance between the electronicdevice and the object.
 16. The method of claim 14, wherein thedetermining the image processing technique comprises: recognizingconfiguration information of the object based on a depth map; andsetting an image filter for processing an image based on theconfiguration information.
 17. The method of claim 14, wherein thedetermining the image processing technique comprises: recognizingconfiguration information based on a contour line of the image data; andsetting an image filter for processing an image based on theconfiguration information.
 18. The method of claim 16, wherein thesetting the image filter sets at least one of: a linear filter, acircular filter, a lighting filter, a macro filter, and a selectivefocus filter based on the recognized configuration information of theobject.
 19. The method of claim 14, wherein the generating the new imagegenerates an image that is blur-processed in its peripheral region witha criterion of a focus of the image filter.
 20. The method of claim 14,wherein the determining the image processing technique further comprisesdisplaying a preview image, and displaying a preview image comprisesdisplaying a preview image to which a dynamic filter being based on ablur and/or exposure change is applied.